Loans - Recorded Investment in Nonaccrual, Restructured and Loans Past Due 90 Days or More and Accruing (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	$ 112,315	$ 137,339	$ 187,135
Nonaccrual Loans	135,216	155,536
Accruing Troubled Debt Restructurings	18,809	29,946
Loans Past Due 90 Days or More and Accruing	1,727	3,030
Total Nonperforming Loans	155,752	188,512
Impaired Financing Receivable, Average Recorded Investment	124,089	164,164	214,045
Impaired Financing Receivable, Interest Income, Cash Basis Method	2,640	2,813	1,592
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	20,727	22,587	40,621
Nonaccrual Loans	20,633	17,324
Accruing Troubled Debt Restructurings	107	5,277
Loans Past Due 90 Days or More and Accruing	80	37
Total Nonperforming Loans	20,820	22,638
Impaired Financing Receivable, Average Recorded Investment	20,523	35,305	23,518
Impaired Financing Receivable, Interest Income, Cash Basis Method	412	529	209
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	41,822	44,278	51,978
Nonaccrual Loans	39,588	40,983
Accruing Troubled Debt Restructurings	2,234	3,295
Loans Past Due 90 Days or More and Accruing	2	1,007
Total Nonperforming Loans	41,824	45,285
Impaired Financing Receivable, Average Recorded Investment	41,426	44,541	49,927
Impaired Financing Receivable, Interest Income, Cash Basis Method	1,151	968	829
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	4,777
Nonaccrual Loans	4,777
Accruing Troubled Debt Restructurings	0
Loans Past Due 90 Days or More and Accruing	0
Total Nonperforming Loans	4,777
Impaired Financing Receivable, Average Recorded Investment	8,723
Impaired Financing Receivable, Interest Income, Cash Basis Method
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment		13,260	24,328
Nonaccrual Loans		13,939
Accruing Troubled Debt Restructurings		0
Loans Past Due 90 Days or More and Accruing		0
Total Nonperforming Loans		13,939
Impaired Financing Receivable, Average Recorded Investment		17,277	58,792
Impaired Financing Receivable, Interest Income, Cash Basis Method
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	10,782	21,574	25,912
Nonaccrual Loans	10,476	14,977
Accruing Troubled Debt Restructurings	306	6,597
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	10,782	21,574
Impaired Financing Receivable, Average Recorded Investment	17,829	27,774	29,152
Impaired Financing Receivable, Interest Income, Cash Basis Method	616	818	339
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	87	158
Accruing Troubled Debt Restructurings	97	100
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	184	258
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	39	149
Accruing Troubled Debt Restructurings	192	175
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	231	324
Commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	33,408	35,622	44,276
Nonaccrual Loans	32,495	33,961
Accruing Troubled Debt Restructurings	913	1,661
Loans Past Due 90 Days or More and Accruing	0	94
Total Nonperforming Loans	33,408	35,716
Impaired Financing Receivable, Average Recorded Investment	34,972	39,248	52,640
Impaired Financing Receivable, Interest Income, Cash Basis Method	461	497	214
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	20,564	28,260
Accruing Troubled Debt Restructurings	11,708	9,425
Loans Past Due 90 Days or More and Accruing	549	950
Total Nonperforming Loans	32,821	38,635
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	2,129	1,689
Accruing Troubled Debt Restructurings	751	736
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	2,880	2,425
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	965	1,670
Accruing Troubled Debt Restructurings	885	780
Loans Past Due 90 Days or More and Accruing	80	54
Total Nonperforming Loans	1,930	2,504
Consumer
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	799	18	20
Nonaccrual Loans	3,463	2,426
Accruing Troubled Debt Restructurings	1,616	1,900
Loans Past Due 90 Days or More and Accruing	1,016	888
Total Nonperforming Loans	6,095	5,214
Impaired Financing Receivable, Average Recorded Investment	616	19	16
Impaired Financing Receivable, Interest Income, Cash Basis Method		$ 1	$ 1