Fair Value Measurements (Details 2) - USD ($) $ in Thousands	9 Months Ended
	Sep. 30, 2016	Dec. 31, 2015
Level 3 CMO Roll Forward [Abstract]
Equity warrants		$ 4,914
Level 3
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation [Line Items]
Warrants, Acquired From Acquisition	$ 4,914
Level 3 CMO Roll Forward [Abstract]
Gain on Sale of Investments	301
Proceeds from Issuance of Warrants	(1,597)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	1,490
Transfers to Available-For-Sale	0
Equity warrants	5,108
Level 3 | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Equity warrants	5,108	$ 4,914
Level 3 | Private Label CMOs | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2015	81,241
Total included in earnings	1,252
Total included in other comprehensive income	(1,399)
Net settlements	(18,273)
Balance, September 30, 2016	62,821
Level 3 | Asset-backed and other securities | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2015	18,200
Total included in earnings	47
Total included in other comprehensive income	(5)
Net settlements	(3,218)
Balance, September 30, 2016	$ 15,024
Level 3 | Minimum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	0.00%
Annual default rates	0.00%
Loss severity rates	0.00%
Discount rates	0.90%
Level 3 | Minimum | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	5.00%
Annual default rates	1.00%
Loss severity rates	10.00%
Discount rates	2.10%
Level 3 | Maximum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	27.60%
Annual default rates	15.00%
Loss severity rates	94.50%
Discount rates	15.00%
Level 3 | Maximum | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	40.00%
Annual default rates	8.00%
Loss severity rates	91.00%
Discount rates	7.20%
Level 3 | Weighted Average | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	11.50%
Annual default rates	2.10%
Loss severity rates	33.10%
Discount rates	3.60%
Level 3 | Weighted Average | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Voluntary annual prepayment speeds	14.70%
Annual default rates	3.70%
Loss severity rates	54.30%
Discount rates	2.90%
Warrant | Level 3
Weighted Average Input
Volatility	19.30%
Warrant | Level 3 | Weighted Average
Weighted Average Input
Risk-free interest rate	1.00%
Remaining life assumption (in years)	3 years 9 months 20 days