Fair Value Measurements (Details 3) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2018	Dec. 31, 2017
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	$ 5,161
Level 3
Level 3 CMO [Roll Forward]
Total included in earnings, Equity Warrants	248
Total included in other comprehensive income	0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	206
Sales, Equity Warrants	(331)
Net settlements, Equity Warrants	0
Fair Value, Measurements, Recurring
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,161
Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	5,161
Private Label CMOs | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	22,081	$ 22,874
Total included in earnings	165
Total included in other comprehensive income	(141)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(804)
Fair value transfers of assets measured on a recurring basis	(13)
Asset-backed | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	36,418	$ 42,109
Total included in earnings	(10)
Total included in other comprehensive income	(228)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(5,453)
Private label collateralized mortgage obligations
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	(13)
SBA securities | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	0
Warrant | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	$ 0