Fair Value Measurements (Details 3) - USD ($) $ in Thousands	6 Months Ended
	Jun. 30, 2018	Dec. 31, 2017
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	$ 5,161
Level 3
Level 3 CMO [Roll Forward]
Total included in earnings, Equity Warrants	1,474
Total included in other comprehensive income	0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	326
Sales, Equity Warrants	(1,745)
Net settlements, Equity Warrants	0
Fair Value, Measurements, Recurring
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,161
Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,161
Private Label CMOs | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	13,751	$ 22,874
Total included in earnings	376
Total included in other comprehensive income	(272)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(9,227)
Asset-backed | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	32,833	$ 42,109
Total included in earnings	(21)
Total included in other comprehensive income	(239)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(9,016)
Private label collateralized mortgage obligations
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	0
SBA securities
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	0
Warrant
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	$ 48