Portfolio Loans - Summary of Recorded Investment in Impaired Portfolio Loans by Category (Details) - Non-Covered Loans - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financing Receivable, Past Due [Line Items]
Non-accrual	$ 26,096	$ 16,520
Restructured, not on non-accrual	79	225
Loans over 90 days past due and still accruing interest	250
Total Recorded Investment	26,425	16,745
Commercial and industrial
Financing Receivable, Past Due [Line Items]
Non-accrual	22,328	12,805
Restructured, not on non-accrual	0	145
Loans over 90 days past due and still accruing interest	250
Total Recorded Investment	22,578	12,950
CRE - investor owned
Financing Receivable, Past Due [Line Items]
Non-accrual	2,303	398
Restructured, not on non-accrual	0	0
Loans over 90 days past due and still accruing interest	0
Total Recorded Investment	2,303	398
CRE - owner occupied
Financing Receivable, Past Due [Line Items]
Non-accrual	213	808
Restructured, not on non-accrual	0	0
Loans over 90 days past due and still accruing interest	0
Total Recorded Investment	213	808
Residential real estate
Financing Receivable, Past Due [Line Items]
Non-accrual	1,251	2,197
Restructured, not on non-accrual	79	80
Loans over 90 days past due and still accruing interest	0
Total Recorded Investment	1,330	2,277
Consumer and other
Financing Receivable, Past Due [Line Items]
Non-accrual	1	312
Restructured, not on non-accrual	0	0
Loans over 90 days past due and still accruing interest	0
Total Recorded Investment	$ 1	$ 312