Portfolio Loans - Summary of Allowance for Loan Losses by Portfolio Class and Category (Details) - Non-Covered Loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	$ 37,565	$ 33,441	$ 30,185
Provision (provision reversal)	10,764	5,551	4,872
Losses charged off	(11,507)	(4,335)	(6,091)
Recoveries	1,344	2,908	4,475
Balance, end of year	38,166	37,565	33,441
Commercial and industrial
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	26,996	22,056	16,983
Provision (provision reversal)	8,737	6,569	6,976
Losses charged off	(9,872)	(2,303)	(3,699)
Recoveries	545	674	1,796
Balance, end of year	26,406	26,996	22,056
CRE - investor owned
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	3,420	3,484	4,382
Provision (provision reversal)	456	(11)	(303)
Losses charged off	(117)	(95)	(664)
Recoveries	131	42	69
Balance, end of year	3,890	3,420	3,484
CRE - owner occupied
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	2,890	2,969	3,135
Provision (provision reversal)	404	(1,202)	(1,626)
Losses charged off	(90)	0	(38)
Recoveries	104	1,123	1,498
Balance, end of year	3,308	2,890	2,969
Construction and land development
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	1,304	1,704	1,715
Provision (provision reversal)	336	(1,334)	(335)
Losses charged off	(254)	0	(350)
Recoveries	101	934	674
Balance, end of year	1,487	1,304	1,704
Residential real estate
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	2,023	1,796	2,830
Provision (provision reversal)	797	129	(58)
Losses charged off	(973)	(25)	(1,313)
Recoveries	390	123	337
Balance, end of year	2,237	2,023	1,796
Consumer and other
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	932	1,432	1,140
Provision (provision reversal)	34	1,400	218
Losses charged off	(201)	(1,912)	(27)
Recoveries	73	12	101
Balance, end of year	$ 838	$ 932	$ 1,432