Loans - Loans Individually and Collectively Evaluated for Impairment (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	$ 106,050	$ 120,975
Loans individually evaluated for impairment	70,426	80,664
Loans collectively evaluated for impairment	35,624	40,311
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	20,657	30,623
Loans individually evaluated for impairment	20,624	30,595
Loans collectively evaluated for impairment	33	28
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	24,474	18,025
Loans individually evaluated for impairment	24,474	18,025
Loans collectively evaluated for impairment	0	0
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	0	2,044
Loans individually evaluated for impairment	0	2,044
Loans collectively evaluated for impairment	0	0
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	2,226	4,676
Loans individually evaluated for impairment	2,226	4,676
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	104	117
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	104	117
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	156	178
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	156	178
Commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	23,102	25,324
Loans individually evaluated for impairment	23,102	25,324
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	27,925	30,521
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	27,925	30,521
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	2,456	2,622
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	2,456	2,622
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	1,253	2,292
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	1,253	2,292
Consumer
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing TDRs	3,697	4,553
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	$ 3,697	$ 4,553