Loans - Loans Individually and Collectively Evaluated for Impairment (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	$ 120,975	$ 116,725
Loans individually evaluated for impairment	80,664	73,712
Loans collectively evaluated for impairment	40,311	43,013
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	30,623	19,123
Loans individually evaluated for impairment	30,595	19,106
Loans collectively evaluated for impairment	28	17
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	18,025	21,989
Loans individually evaluated for impairment	18,025	21,989
Loans collectively evaluated for impairment	0	0
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	2,044	2,078
Loans individually evaluated for impairment	2,044	2,078
Loans collectively evaluated for impairment	0	0
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	4,676	5,609
Loans individually evaluated for impairment	4,676	5,609
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	117	153
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	117	153
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	178	240
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	178	240
Commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	25,324	24,930
Loans individually evaluated for impairment	25,324	24,930
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	30,521	32,218
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	30,521	32,218
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	2,622	2,509
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	2,622	2,509
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	2,292	2,522
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	2,292	2,522
Consumer
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	4,553	5,354
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	$ 4,553	$ 5,354