Investment Securities (Unrealized Gain (Loss) on Investments) (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Net Investment Income [Line Items]
Amortized cost	$ 469,290	$ 496,536
Gross unrealized gains	3,226	2,847
Gross unrealized losses	(2,796)	(4,564)
Estimated fair value	469,720	494,819
Amortized cost	139,196	140,557
Gross unrealized gains	2,607	1,965
Gross unrealized losses	(209)	(250)
Estimated fair value	141,594	142,272
U.S. Agency mortgage-backed securities (MBS)
Net Investment Income [Line Items]
Amortized cost	158,881	165,933
Gross unrealized gains	1,939	1,828
Gross unrealized losses	(317)	(799)
Estimated fair value	160,503	166,962
Amortized cost	100,305	101,563
Gross unrealized gains	1,606	1,195
Gross unrealized losses	(110)	(148)
Estimated fair value	101,801	102,610
Non-agency MBS
Net Investment Income [Line Items]
Amortized cost	232,213	252,153
Gross unrealized gains	271	203
Gross unrealized losses	(2,202)	(3,408)
Estimated fair value	230,282	248,948
U.S. Agency asset-backed securities
Net Investment Income [Line Items]
Amortized cost	6,424	6,700
Gross unrealized gains	416	419
Gross unrealized losses	(62)	(57)
Estimated fair value	6,778	7,062
Corporate securities
Net Investment Income [Line Items]
Amortized cost	71,232	71,213
Gross unrealized gains	583	397
Gross unrealized losses	(215)	(287)
Estimated fair value	71,600	71,323
Mutual fund
Net Investment Income [Line Items]
Amortized cost	540	537
Gross unrealized gains	17	0
Gross unrealized losses	0	(13)
Estimated fair value	557	524
Obligations of state and political subdivisions
Net Investment Income [Line Items]
Amortized cost	38,604	287
Gross unrealized gains	1,001	0
Gross unrealized losses	(99)	0
Estimated fair value	39,506	287
Tax credit investments
Net Investment Income [Line Items]
Amortized cost	287	38,707
Gross unrealized gains	0	770
Gross unrealized losses	0	(102)
Estimated fair value	$ 287	$ 39,375