Fair Value Measurements (Details 3) - USD ($)	3 Months Ended
	Mar. 31, 2020	Dec. 31, 2019	Mar. 31, 2019
Level 3 CMO [Roll Forward]
Unrealized net holding gains (losses) on securities available-for-sale arising during the period	$ 17,183,000	$ (24,547,000)	$ 62,639,000
Level 3
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	3,434,000
Total included in earnings, Equity Warrants	837,000
Total included in other comprehensive income	0
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	120,000
Sales, Equity Warrants	(101,000)
Net settlements, Equity Warrants	0
Equity warrants, Balance, end of period		3,434,000
Fair Value, Measurements, Recurring
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	3,434,000
Equity warrants, Balance, end of period	4,290,000	3,434,000
Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Equity warrants, Balance, beginning of period	3,434,000
Equity warrants, Balance, end of period	4,290,000	3,434,000
Private Label CMOs | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	6,264,000
Total included in earnings	118,000
Total included in other comprehensive income	(959,000)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(481,000)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	4,942,000	6,264,000
Unrealized net holding gains (losses) on securities available-for-sale arising during the period	726,000
Asset-backed | Fair Value, Measurements, Recurring | Level 3
Level 3 CMO [Roll Forward]
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	16,435,000
Total included in earnings	(11,000)
Total included in other comprehensive income	(687,000)
Stock and Warrants Issued During Period, Value, Preferred Stock and Warrants	0
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Sales	0
Net settlements	(1,118,000)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset Value	14,619,000	$ 16,435,000
Unrealized net holding gains (losses) on securities available-for-sale arising during the period	(490,000)
Warrant
Level 3 CMO [Roll Forward]
Fair value transfers of assets measured on a recurring basis	$ 0