Fair Value Measurements (Details 2) - USD ($) $ in Thousands	6 Months Ended
	Jun. 30, 2016	Dec. 31, 2015
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation [Line Items]
Warrants, Acquired From Acquisition	$ 4,914
Level 3 CMO Roll Forward [Abstract]
Gain on Sale of Investments	88
Proceeds from Issuance of Warrants	922
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	1,112
Warrants Not Settleable in Cash, Fair Value Disclosure	5,192	$ 4,914
Transfers to Available-For-Sale	0
Level 3 | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Warrants Not Settleable in Cash, Fair Value Disclosure	5,192	$ 4,914
Level 3 | Private Label CMOs | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2015	81,241
Total included in earnings	830
Total included in other comprehensive income	(1,409)
Net settlements	(11,632)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	69,030
Level 3 | Asset-backed and other securities | Fair Value, Measurements, Recurring
Level 3 CMO Roll Forward [Abstract]
Balance, December 31, 2015	18,200
Total included in earnings	31
Total included in other comprehensive income	12
Net settlements	(2,107)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	$ 16,136
Level 3 | Minimum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	0.90%
Fair Value Inputs, Probability of Default	0.20%
Fair Value Inputs, Loss Severity	5.40%
Fair Value Inputs, Discount Rate	0.80%
Level 3 | Minimum | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	5.00%
Fair Value Inputs, Probability of Default	1.00%
Fair Value Inputs, Loss Severity	10.00%
Fair Value Inputs, Discount Rate	1.70%
Level 3 | Maximum | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	24.47%
Fair Value Inputs, Probability of Default	14.73%
Fair Value Inputs, Loss Severity	94.08%
Fair Value Inputs, Discount Rate	65.75%
Level 3 | Maximum | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	40.00%
Fair Value Inputs, Probability of Default	8.00%
Fair Value Inputs, Loss Severity	91.00%
Fair Value Inputs, Discount Rate	4.70%
Level 3 | Weighted Average | Private label collateralized mortgage obligations | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	11.60%
Fair Value Inputs, Probability of Default	1.90%
Fair Value Inputs, Loss Severity	23.20%
Fair Value Inputs, Discount Rate	3.70%
Level 3 | Weighted Average | SBA securities | Fair Value, Measurements, Recurring
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Inputs, Prepayment Rate	14.20%
Fair Value Inputs, Probability of Default	3.70%
Fair Value Inputs, Loss Severity	52.40%
Fair Value Inputs, Discount Rate	2.80%
Warrant [Member] | Level 3
Unobservable Inputs Quantitative Information [Abstract]
Fair Value Assumptions, Weighted Average Volatility Rate	23.80%
Warrant [Member] | Level 3 | Weighted Average
Fair Value Assumptions [Abstract]
Fair Value Assumptions, Risk Free Interest Rate	0.80%
Fair Value Assumptions, Remaining life assumption	3 years 8 months 18 days