Loans - Recorded Investment in Nonaccrual, Restructured and Loans Past Due 90 Days or More and Accruing (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2014	Dec. 31, 2013	Dec. 31, 2012
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	$ 73,712	$ 112,315	$ 137,339
Nonaccrual Loans	100,393	135,216
Accruing Troubled Debt Restructurings	16,332	18,809
Loans Past Due 90 Days or More and Accruing	2,709	1,727
Total Nonperforming Loans	119,434	155,752
Impaired Financing Receivable, Average Recorded Investment	92,614	124,089	164,164
Impaired Financing Receivable, Interest Income, Cash Basis Method	2,678	2,640	2,813
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	19,106	20,727	22,587
Nonaccrual Loans	18,826	20,633
Accruing Troubled Debt Restructurings	297	107
Loans Past Due 90 Days or More and Accruing	229	80
Total Nonperforming Loans	19,352	20,820
Impaired Financing Receivable, Average Recorded Investment	19,518	20,523	35,305
Impaired Financing Receivable, Interest Income, Cash Basis Method	360	412	529
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	21,989	41,822	44,278
Nonaccrual Loans	19,299	39,588
Accruing Troubled Debt Restructurings	2,690	2,234
Loans Past Due 90 Days or More and Accruing	0	2
Total Nonperforming Loans	21,989	41,824
Impaired Financing Receivable, Average Recorded Investment	31,945	41,426	44,541
Impaired Financing Receivable, Interest Income, Cash Basis Method	1,027	1,151	968
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	2,078	4,777	13,260
Nonaccrual Loans	2,078
Accruing Troubled Debt Restructurings	0
Loans Past Due 90 Days or More and Accruing	0
Total Nonperforming Loans	2,078
Impaired Financing Receivable, Average Recorded Investment	3,658	8,723	17,277
Impaired Financing Receivable, Interest Income, Cash Basis Method	146
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans		4,777
Accruing Troubled Debt Restructurings		0
Loans Past Due 90 Days or More and Accruing		0
Total Nonperforming Loans		4,777
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	5,609	10,782	21,574
Nonaccrual Loans	5,558	10,476
Accruing Troubled Debt Restructurings	51	306
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	5,609	10,782
Impaired Financing Receivable, Average Recorded Investment	8,784	17,829	27,774
Impaired Financing Receivable, Interest Income, Cash Basis Method	61	616	818
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	59	87
Accruing Troubled Debt Restructurings	94	97
Loans Past Due 90 Days or More and Accruing	9	0
Total Nonperforming Loans	162	184
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	115	39
Accruing Troubled Debt Restructurings	125	192
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	240	231
Commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	24,930	33,408	35,622
Nonaccrual Loans	24,336	32,495
Accruing Troubled Debt Restructurings	594	913
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	24,930	33,408
Impaired Financing Receivable, Average Recorded Investment	28,306	34,972	39,248
Impaired Financing Receivable, Interest Income, Cash Basis Method	1,084	461	497
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	21,869	20,564
Accruing Troubled Debt Restructurings	10,349	11,708
Loans Past Due 90 Days or More and Accruing	1,329	549
Total Nonperforming Loans	33,547	32,821
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	1,879	2,129
Accruing Troubled Debt Restructurings	630	751
Loans Past Due 90 Days or More and Accruing	9	0
Total Nonperforming Loans	2,518	2,880
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	1,743	965
Accruing Troubled Debt Restructurings	779	885
Loans Past Due 90 Days or More and Accruing	0	80
Total Nonperforming Loans	2,522	1,930
Consumer
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment		799	18
Nonaccrual Loans	4,631	3,463
Accruing Troubled Debt Restructurings	723	1,616
Loans Past Due 90 Days or More and Accruing	1,133	1,016
Total Nonperforming Loans	6,487	6,095
Impaired Financing Receivable, Average Recorded Investment	403	616	19
Impaired Financing Receivable, Interest Income, Cash Basis Method			$ 1