Investment Securities (Details) (USD $) In Thousands, unless otherwise specified	Jun. 30, 2014	Dec. 31, 2013
Available-for-sale Securities [Abstract]
Amortized cost	$ 409,715	$ 438,057	[1]
Gross unrealized gains	2,456	3,048
Gross unrealized losses	13,061	26,491
Estimated fair value	399,110	414,614	[1]
Held-to-maturity Securities [Abstract]
Amortized cost	147,055	151,795
Gross unrealized gains	0	0
Gross unrealized losses	4,767	10,670
Estimated fair value	142,288	141,125	[1]
Amortized cost, Total	556,770	589,852
Gross unrealized gains, Total	2,456	3,048
Gross unrealized losses, Total	17,828	37,161
Estimated fair value, Total	541,398	555,739
Obligations of U.S. government sponsored enterprises [Member]
Available-for-sale Securities [Abstract]
Amortized cost	2,040	2,051
Gross unrealized gains	24	26
Gross unrealized losses	0	0
Estimated fair value	2,064	2,077
Residential mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	347,916	364,513
Gross unrealized gains	1,278	974
Gross unrealized losses	12,298	24,340
Estimated fair value	336,896	341,147
Held-to-maturity Securities [Abstract]
Amortized cost	136,987	141,724
Gross unrealized gains	0	0
Gross unrealized losses	4,589	9,907
Estimated fair value	132,398	131,817
Residential mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	18,404	19,770
Gross unrealized gains	1,104	982
Gross unrealized losses	0	0
Estimated fair value	19,508	20,752
Commercial mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	22,573	22,767
Gross unrealized gains	0	0
Gross unrealized losses	522	1,328
Estimated fair value	22,051	21,439
Commercial mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	10,387	10,408
Gross unrealized gains	0	0
Gross unrealized losses	241	823
Estimated fair value	10,146	9,585
Held-to-maturity Securities [Abstract]
Amortized cost	10,068	10,071
Gross unrealized gains	0	0
Gross unrealized losses	178	763
Estimated fair value	9,890	9,308
Business Development Company investment [Member]
Available-for-sale Securities [Abstract]
Amortized cost	0	1,753
Gross unrealized gains	0	984
Gross unrealized losses	0
Estimated fair value	0	2,737
Corporate notes [Member]
Available-for-sale Securities [Abstract]
Amortized cost	8,395	16,795
Gross unrealized gains	50	82
Gross unrealized losses	0	0
Estimated fair value	$ 8,445	$ 16,877

[1] Derived from audited consolidated financial statements