Investment Securities (Details) (USD $) In Thousands, unless otherwise specified	Mar. 31, 2014	Dec. 31, 2013
Available-for-sale Securities [Abstract]
Amortized cost	$ 420,619	$ 438,057	[1]
Gross unrealized gains	2,921	3,048
Gross unrealized losses	21,072	26,491
Estimated fair value	402,468	414,614	[1]
Held-to-maturity Securities [Abstract]
Amortized cost	149,060	151,795
Gross unrealized gains	0	0
Gross unrealized losses	8,058	10,670
Estimated fair value	141,002	141,125	[1]
Amortized cost, Total	569,679	589,852
Gross unrealized gains, Total	2,921	3,048
Gross unrealized losses, Total	29,130	37,161
Estimated fair value, Total	543,470	555,739
Obligations of U.S. government sponsored enterprises [Member]
Available-for-sale Securities [Abstract]
Amortized cost	2,045	2,051
Gross unrealized gains	25	26
Gross unrealized losses	0	0
Estimated fair value	2,070	2,077
Residential mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	356,022	364,513
Gross unrealized gains	971	974
Gross unrealized losses	19,429	24,340
Estimated fair value	337,564	341,147
Held-to-maturity Securities [Abstract]
Amortized cost	138,991	141,724
Gross unrealized gains	0	0
Gross unrealized losses	7,564	9,907
Estimated fair value	131,427	131,817
Residential mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	19,338	19,770
Gross unrealized gains	953	982
Gross unrealized losses	9	0
Estimated fair value	20,282	20,752
Commercial mortgage-backed securities - GSE [Member]
Available-for-sale Securities [Abstract]
Amortized cost	22,670	22,767
Gross unrealized gains	0	0
Gross unrealized losses	1,080	1,328
Estimated fair value	21,590	21,439
Commercial mortgage-backed securities - Private [Member]
Available-for-sale Securities [Abstract]
Amortized cost	10,398	10,408
Gross unrealized gains	0	0
Gross unrealized losses	554	823
Estimated fair value	9,844	9,585
Held-to-maturity Securities [Abstract]
Amortized cost	10,069	10,071
Gross unrealized gains	0	0
Gross unrealized losses	494	763
Estimated fair value	9,575	9,308
Business Development Company investment [Member]
Available-for-sale Securities [Abstract]
Amortized cost	1,753	1,753
Gross unrealized gains	898	984
Gross unrealized losses	0
Estimated fair value	2,651	2,737
Corporate notes [Member]
Available-for-sale Securities [Abstract]
Amortized cost	8,393	16,795
Gross unrealized gains	74	82
Gross unrealized losses	0	0
Estimated fair value	$ 8,467	$ 16,877

[1] Derived from audited consolidated financial statements