Portfolio Loans - Summary of Recorded Investment in Impaired Non-covered Loans by Category (Details) (Non-Covered Loans, USD $) In Thousands, unless otherwise specified	Sep. 30, 2014	Dec. 31, 2013
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	$ 17,418	$ 20,888
Restructured	1,364	678
Loans over 90 days past due and still accruing interest	340	0
Total	19,122	21,566
Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	3,221	3,384
Restructured	0	0
Loans over 90 days past due and still accruing interest	340	0
Total	3,561	3,384
Commercial real estate - Investor owned
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	4,755	6,511
Restructured	587	678
Loans over 90 days past due and still accruing interest	0	0
Total	5,342	7,189
Commercial real estate - Owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	2,192	622
Restructured	777	0
Loans over 90 days past due and still accruing interest	0	0
Total	2,969	622
Construction and land development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	6,849	9,802
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total	6,849	9,802
Residential real estate
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	401	569
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total	401	569
Consumer and other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	0	0
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total	$ 0	$ 0