Portfolio Loans - Summary of Recorded Investment in Impaired Portfolio Loans by Category (Details) - Non-Covered Loans - USD ($)	Jun. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	$ 11,377,000	$ 9,680,000
Restructured	2,330,000	307,000
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	13,707,000	9,987,000
Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	2,524,000	4,406,000
Restructured	2,310,000	307,000
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0.000	0.000
Total Recorded Investment	4,834,000	4,713,000
Commercial - investor owned
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	249,000	927,000
Restructured	0	0
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	249,000	927,000
Commercial - owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	0	198,000
Restructured	0	0
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	0	198,000
Construction and land development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	3,317,000	3,444,000
Restructured	20,000	0
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	3,337,000	3,444,000
Residential real estate
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	707,000	705,000
Restructured	0	0
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	707,000	705,000
Consumer and other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	4,580,000	0
Restructured	0	0
Impaired Financing Receivable, with No Related Allowance, Unpaid Principal Balance	0	0
Total Recorded Investment	$ 4,580,000	$ 0