Fair Value Measurements (Details 1) - USD ($) $ in Thousands	6 Months Ended
	Jun. 30, 2017	Jun. 30, 2016
Level 3 CMO Roll Forward [Abstract]
Equity warrants, Balance, beginning of period	$ 5,497
Fair value transfers of assets measured on a recurring basis	0
Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Equity warrants, Balance, beginning of period	5,497
Equity warrants, Balance, end of period	5,452
Level 3
Level 3 CMO Roll Forward [Abstract]
Total included in earnings, Equity Warrants	969
Total included in other comprehensive income	0
Purchases		$ 0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	535
Sales, Equity Warrants	(1,549)
Net settlements, Equity Warrants	0
Level 3 | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Equity warrants, Balance, beginning of period	5,497
Equity warrants, Balance, end of period	5,452
Level 3 | Private Label CMOs | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2016	56,902
Total included in earnings	2,230
Total included in other comprehensive income	(887)
Purchases		18,643
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	(4,732)
Net settlements	(12,617)
Balance, June 30, 2017	59,539
Level 3 | Asset-backed and other securities | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2016	8,373
Total included in earnings	59
Total included in other comprehensive income	(75)
Purchases		$ 0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(1,594)
Balance, June 30, 2017	$ 6,763
Level 3 | Minimum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	2.70%
Annual default rates	0.10%
Loss severity rates	3.80%
Discount rates	1.40%
Level 3 | Maximum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	23.90%
Annual default rates	14.10%
Loss severity rates	132.70%
Discount rates	9.80%
Level 3 | Weighted Average | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	11.50%
Annual default rates	2.20%
Loss severity rates	50.40%
Discount rates	4.20%
Level 3 | Weighted Average | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	5.00%
Annual default rates	1.00%
Loss severity rates	10.00%
Discount rates	3.50%
Level 3 | Weighted Average | Warrant | Fair Value, Measurements, Recurring
Weighted Average Input
Remaining life assumption (in years)	3 years 8 months
Warrant | Level 3
Weighted Average Input
Volatility	17.30%
Warrant | Level 3 | Weighted Average
Weighted Average Input
Risk-free interest rate	1.70%