Fair Value Measurements - Estimate of the Fair Value of Financial Instruments Not Recorded at Fair Value on Balance Sheet (Details) (USD $) In Thousands, unless otherwise specified	Jun. 30, 2014	Dec. 31, 2013
Financial Assets:
State tax credits held for sale	$ 14,985	$ 16,491
Fair Value, Level 1
Financial Assets:
Portfolio loans, net	0	0
State tax credits held for sale	0	0
Financial Liabilities:
Deposits	1,867,330	1,902,038
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Fair Value, Level 2
Financial Assets:
Portfolio loans, net	0	0
State tax credits held for sale	0	0
Financial Liabilities:
Deposits	0	0
Subordinated debentures	33,857	39,358
Federal Home Loan Bank advances	157,155	54,137
Other borrowings	172,261	214,377
Fair Value, Level 3
Financial Assets:
Portfolio loans, net	2,319,524	2,232,134
State tax credits held for sale	34,849	35,668
Financial Liabilities:
Deposits	603,193	638,784
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Estimated fair value
Financial Assets:
Portfolio loans, net	2,319,524	2,232,134
State tax credits held for sale	34,849	35,668
Financial Liabilities:
Deposits	2,470,523	2,540,822
Subordinated debentures	33,857	39,358
Federal Home Loan Bank advances	157,155	54,137
Other borrowings	$ 172,261	$ 214,377