Loans - Loans Individually and Collectively Evaluated for Impairment (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2013	Dec. 31, 2012
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	$ 154,025	$ 185,482
Loans individually evaluated for impairment	112,315	137,339
Loans collectively evaluated for impairment	41,710	48,143
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	20,740	22,601
Loans individually evaluated for impairment	20,727	22,587
Loans collectively evaluated for impairment	13	14
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	41,822	44,278
Loans individually evaluated for impairment	41,822	44,278
Loans collectively evaluated for impairment	0	0
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	4,777	13,939
Loans individually evaluated for impairment	4,777	13,260
Loans collectively evaluated for impairment	0	679
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	10,782	21,574
Loans individually evaluated for impairment	10,782	21,574
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	184	258
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	184	258
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	231	324
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	231	324
Commercial
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	33,408	35,622
Loans individually evaluated for impairment	33,408	35,622
Loans collectively evaluated for impairment	0	0
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	32,272	37,685
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	32,272	37,685
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	2,880	2,425
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	2,880	2,425
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	1,850	2,450
Loans individually evaluated for impairment	0	0
Loans collectively evaluated for impairment	1,850	2,450
Consumer
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual and accruing troubled debt restructurings	5,079	4,326
Loans individually evaluated for impairment	799	18
Loans collectively evaluated for impairment	$ 4,280	$ 4,308