Loans - Recorded Investment in Nonaccrual, Restructured and Loans Past Due 90 Days or More and Accruing (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	$ 80,664	$ 73,712	$ 112,315
Nonaccrual Loans	95,887	100,393
Accruing Troubled Debt Restructurings	25,088	16,332
Loans Past Due 90 Days or More and Accruing	1,975	2,709
Total Nonperforming Loans	122,950	119,434
Impaired Financing Receivable, Average Recorded Investment	70,762	92,614	124,089
Impaired Financing Receivable, Interest Income, Cash Basis Method	1,963	2,678	2,640
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	30,595	19,106	20,727
Nonaccrual Loans	21,676	18,826
Accruing Troubled Debt Restructurings	8,947	297
Loans Past Due 90 Days or More and Accruing	0	229
Total Nonperforming Loans	30,623	19,352
Impaired Financing Receivable, Average Recorded Investment	20,179	19,518	20,523
Impaired Financing Receivable, Interest Income, Cash Basis Method	340	360	412
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	18,025	21,989	41,822
Nonaccrual Loans	15,268	19,299
Accruing Troubled Debt Restructurings	2,757	2,690
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	18,025	21,989
Impaired Financing Receivable, Average Recorded Investment	17,883	31,945	41,426
Impaired Financing Receivable, Interest Income, Cash Basis Method	550	1,027	1,151
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	2,044	2,078	4,777
Nonaccrual Loans	2,044	2,078
Accruing Troubled Debt Restructurings	0	0
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	2,044	2,078
Impaired Financing Receivable, Average Recorded Investment	2,066	3,658	8,723
Impaired Financing Receivable, Interest Income, Cash Basis Method	21	146
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	4,676	5,609	10,782
Nonaccrual Loans	4,162	5,558
Accruing Troubled Debt Restructurings	514	51
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	4,676	5,609
Impaired Financing Receivable, Average Recorded Investment	5,666	8,784	17,829
Impaired Financing Receivable, Interest Income, Cash Basis Method	26	61	616
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	7	59
Accruing Troubled Debt Restructurings	110	94
Loans Past Due 90 Days or More and Accruing	0	9
Total Nonperforming Loans	117	162
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	64	115
Accruing Troubled Debt Restructurings	114	125
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	178	240
Commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	25,324	24,930	33,408
Nonaccrual Loans	25,063	24,336
Accruing Troubled Debt Restructurings	261	594
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	25,324	24,930
Impaired Financing Receivable, Average Recorded Investment	24,968	28,306	34,972
Impaired Financing Receivable, Interest Income, Cash Basis Method	1,026	1,084	461
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	20,378	21,869
Accruing Troubled Debt Restructurings	10,143	10,349
Loans Past Due 90 Days or More and Accruing	851	1,329
Total Nonperforming Loans	31,372	33,547
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	1,749	1,879
Accruing Troubled Debt Restructurings	873	630
Loans Past Due 90 Days or More and Accruing	27	9
Total Nonperforming Loans	2,649	2,518
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	1,657	1,743
Accruing Troubled Debt Restructurings	635	779
Loans Past Due 90 Days or More and Accruing	4	0
Total Nonperforming Loans	2,296	2,522
Consumer
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment			799
Nonaccrual Loans	3,819	4,631
Accruing Troubled Debt Restructurings	734	723
Loans Past Due 90 Days or More and Accruing	1,093	1,133
Total Nonperforming Loans	$ 5,646	6,487
Impaired Financing Receivable, Average Recorded Investment		$ 403	$ 616