Portfolio Loans - Summary of Allowance for Loan Losses by Portfolio Class and Category (Details) (Non-Covered Loans, USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2014	Dec. 31, 2013	Dec. 31, 2012
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	$ 27,289	$ 34,330	$ 37,989
Provision charged to expense	4,409	(642)	8,757
Losses charged off	(5,556)	(10,378)	(15,276)
Recoveries	4,043	3,979	2,860
Balance, end of year	30,185	27,289	34,330
Commercial & Industrial
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	12,246	10,064	11,945
Provision charged to expense	6,728	3,810	774
Losses charged off	(3,738)	(3,404)	(3,233)
Recoveries	1,768	1,776	578
Balance, end of year	17,004	12,246	10,064
Commercial Real Estate Owner Occupied
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	4,096	4,192	6,297
Provision charged to expense	(1,027)	410	1,173
Losses charged off	(450)	(550)	(3,326)
Recoveries	1,006	44	48
Balance, end of year	3,625	4,096	4,192
Commercial Real Estate Investor Owned
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	6,600	10,403	6,751
Provision charged to expense	(1,847)	(94)	6,294
Losses charged off	(250)	(4,441)	(2,728)
Recoveries	95	732	86
Balance, end of year	4,598	6,600	10,403
Construction and Land Development
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	2,136	5,239	5,847
Provision charged to expense	(317)	(2,695)	3,081
Losses charged off	(905)	(896)	(4,384)
Recoveries	806	488	695
Balance, end of year	1,720	2,136	5,239
Residential Real Estate
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	2,019	2,026	3,931
Provision charged to expense	525	107	(1,751)
Losses charged off	(48)	(1,053)	(1,605)
Recoveries	334	939	1,451
Balance, end of year	2,830	2,019	2,026
Consumer & Other
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	192	31	14
Provision charged to expense	347	195	15
Losses charged off	(165)	(34)	0
Recoveries	34	0	2
Balance, end of year	408	192	31
Unallocated
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	0	2,375	3,204
Provision charged to expense	0	(2,375)	(829)
Losses charged off	0	0	0
Recoveries	0	0	0
Balance, end of year	$ 0	$ 0	$ 2,375