Loans - Recorded Investment in Nonaccrual, Restructured and Loans Past Due 90 Days or More and Accruing (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	$ 70,426	$ 80,664	$ 73,712
Nonaccrual Loans	87,822	95,887
Accruing Troubled Debt Restructurings	18,228	25,088
Loans Past Due 90 Days or More and Accruing	2,138	1,975
Total Nonperforming Loans	108,188	122,950
Impaired Financing Receivable, Average Recorded Investment	79,496	70,762	92,614
Impaired Financing Receivable, Interest Income, Cash Basis Method	4,777	1,963	2,678
Commercial, financial and agricultural
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	20,624	30,595	19,106
Nonaccrual Loans	20,057	21,676
Accruing Troubled Debt Restructurings	600	8,947
Loans Past Due 90 Days or More and Accruing	15	0
Total Nonperforming Loans	20,672	30,623
Impaired Financing Receivable, Average Recorded Investment	26,821	20,179	19,518
Impaired Financing Receivable, Interest Income, Cash Basis Method	885	340	360
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	24,474	18,025	21,989
Nonaccrual Loans	19,169	15,268
Accruing Troubled Debt Restructurings	5,305	2,757
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	24,474	18,025
Impaired Financing Receivable, Average Recorded Investment	22,828	17,883	31,945
Impaired Financing Receivable, Interest Income, Cash Basis Method	884	550	1,027
SEPH commercial land and development
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment		2,044	2,078
Nonaccrual Loans	0	2,044
Accruing Troubled Debt Restructurings	0	0
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	0	2,044
Impaired Financing Receivable, Average Recorded Investment	1,597	2,066	3,658
Impaired Financing Receivable, Interest Income, Cash Basis Method		21	146
Remaining commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	2,226	4,676	5,609
Nonaccrual Loans	1,833	4,162
Accruing Troubled Debt Restructurings	393	514
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	2,226	4,676
Impaired Financing Receivable, Average Recorded Investment	3,906	5,666	8,784
Impaired Financing Receivable, Interest Income, Cash Basis Method	66	26	61
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	0	7
Accruing Troubled Debt Restructurings	104	110
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	104	117
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	61	64
Accruing Troubled Debt Restructurings	95	114
Loans Past Due 90 Days or More and Accruing	12	0
Total Nonperforming Loans	168	178
Commercial
Financing Receivable, Recorded Investment [Line Items]
Impaired Financing Receivable, Recorded Investment	23,102	25,324	24,930
Nonaccrual Loans	23,013	25,063
Accruing Troubled Debt Restructurings	89	261
Loans Past Due 90 Days or More and Accruing	0	0
Total Nonperforming Loans	23,102	25,324
Impaired Financing Receivable, Average Recorded Investment	24,341	24,968	28,306
Impaired Financing Receivable, Interest Income, Cash Basis Method	2,942	1,026	1,084
Mortgage
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	18,313	20,378
Accruing Troubled Debt Restructurings	9,612	10,143
Loans Past Due 90 Days or More and Accruing	887	851
Total Nonperforming Loans	28,812	31,372
HELOC
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	1,783	1,749
Accruing Troubled Debt Restructurings	673	873
Loans Past Due 90 Days or More and Accruing	25	27
Total Nonperforming Loans	2,481	2,649
Installment
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	644	1,657
Accruing Troubled Debt Restructurings	609	635
Loans Past Due 90 Days or More and Accruing	60	4
Total Nonperforming Loans	1,313	2,296
Consumer
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual Loans	2,949	3,819
Accruing Troubled Debt Restructurings	748	734
Loans Past Due 90 Days or More and Accruing	1,139	1,093
Total Nonperforming Loans	4,836	$ 5,646
Impaired Financing Receivable, Average Recorded Investment	$ 3		$ 403