Loans - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016	Mar. 31, 2016
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,272,000	$ 1,284,082	$ 1,250,049
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	469,815	459,574	441,160
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	464,859	481,155	467,648
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 934,674	$ 940,729
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 100,248	$ 100,959	84,790
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 159,426	$ 157,872	149,961
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 75,258	$ 82,660	103,181
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	969	1,095	1,590
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,425	767	$ 1,719
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 77,652	$ 84,522
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 354,424	$ 346,246
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	412,694	420,970
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 767,118	$ 767,216
Percentage of Total Loans	82.07%	81.56%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 100,050	$ 100,775
Percentage of Total Loans	99.80%	99.82%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 154,943	$ 153,403
Percentage of Total Loans	97.19%	97.17%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 10,916	$ 22,611
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	14,188	23,085
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 25,104	$ 45,696
Percentage of Total Loans	2.69%	4.86%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 0
Percentage of Total Loans	0.00%	0.00%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,131	$ 1,181
Percentage of Total Loans	0.71%	0.75%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 104,261	$ 90,300
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	37,977	37,100
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 142,238	$ 127,400
Percentage of Total Loans	15.22%	13.54%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 198	$ 184
Percentage of Total Loans	0.20%	0.18%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,352	$ 3,288
Percentage of Total Loans	2.10%	2.08%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 214	$ 417
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 214	$ 417
Percentage of Total Loans	0.02%	0.04%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 74,753	$ 82,103
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	969	1,095
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,425	767
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 77,147	$ 83,965
Percentage of Total Loans	99.35%	99.34%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 505	$ 557
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 505	$ 557
Percentage of Total Loans	0.65%	0.66%