Investment Securities (Details) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014	Dec. 31, 2013
Available-for-sale Securities [Abstract]
Amortized cost	$ 374,457	$ 438,057	[1]
Gross unrealized gains	1,655	3,048
Gross unrealized losses	12,816	26,491
Estimated fair value	363,296	414,614	[1]
Held-to-maturity Securities [Abstract]
Amortized cost	144,684	151,795
Gross unrealized gains	0	0
Gross unrealized losses	5,102	10,670
Estimated fair value	139,582	141,125	[1]
Amortized cost, Total	519,141	589,852
Gross unrealized gains, Total	1,655	3,048
Gross unrealized losses, Total	17,918	37,161
Estimated fair value, Total	502,878	555,739
Obligations of U.S. government sponsored enterprises [Member]
Available-for-sale Securities [Abstract]
Amortized cost	2,034	2,051
Gross unrealized gains	19	26
Gross unrealized losses	0	0
Estimated fair value	2,053	2,077
Residential mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	313,522	364,513
Gross unrealized gains	591	974
Gross unrealized losses	11,666	24,340
Estimated fair value	302,447	341,147
Held-to-maturity Securities [Abstract]
Amortized cost	134,618	141,724
Gross unrealized gains	0	0
Gross unrealized losses	4,859	9,907
Estimated fair value	129,759	131,817
Residential mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	17,653	19,770
Gross unrealized gains	1,015	982
Gross unrealized losses	0	0
Estimated fair value	18,668	20,752
Commercial mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	22,475	22,767
Gross unrealized gains	0	0
Gross unrealized losses	831	1,328
Estimated fair value	21,644	21,439
Commercial mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	10,376	10,408
Gross unrealized gains	0	0
Gross unrealized losses	319	823
Estimated fair value	10,057	9,585
Held-to-maturity Securities [Abstract]
Amortized cost	10,066	10,071
Gross unrealized gains	0	0
Gross unrealized losses	243	763
Estimated fair value	9,823	9,308
Business Development Company investment [Member]
Available-for-sale Securities [Abstract]
Amortized cost		1,753
Gross unrealized gains		984
Estimated fair value		2,737
Corporate notes [Member]
Available-for-sale Securities [Abstract]
Amortized cost	8,397	16,795
Gross unrealized gains	30	82
Gross unrealized losses	0	0
Estimated fair value	$ 8,427	$ 16,877

[1] Derived from audited consolidated financial statements