Fair Value Measurements (Details 3) - USD ($) $ in Thousands	9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Dec. 31, 2016
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	$ 5,497
Fair value transfers of assets measured on a recurring basis	(654)
Level 3
Level 3 CMO [Roll Forward]
Total included in earnings, Equity Warrants	1,701
Total included in other comprehensive income	0
Purchases		$ 0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	720
Sales, Equity Warrants	(2,106)
Net settlements, Equity Warrants	0
Fair Value, Measurements, Recurring
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,497
Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,497
Private Label CMOs | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	37,936		$ 56,902
Total included in earnings	2,114
Total included in other comprehensive income	(578)
Purchases		0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	(4,732)
Net settlements	(15,770)
Asset-backed and other securities | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	42,385		$ 8,373
Total included in earnings	314
Total included in other comprehensive income	(349)
Purchases		$ 37,722
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(3,675)
Private label collateralized mortgage obligations
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	0
SBA securities
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	0
Warrant
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	$ (654)