Loans - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016	Jun. 30, 2016
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,240,253	$ 1,284,082	$ 1,262,389
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	451,767	459,574	456,264
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	461,064	481,155	463,142
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 912,831	$ 940,729
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 98,695	$ 100,959	96,331
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 156,394	$ 157,872	148,379
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 70,031	$ 82,660	94,522
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	866	1,095	1,641
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,436	767	$ 2,110
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 72,333	$ 84,522
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 343,330	$ 346,246
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	409,145	420,970
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 752,475	$ 767,216
Percentage of Total Loans	82.43%	81.56%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 95,865	$ 100,775
Percentage of Total Loans	97.13%	99.82%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 152,236	$ 153,403
Percentage of Total Loans	97.34%	97.17%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 13,330	$ 22,611
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	3,336	23,085
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 16,666	$ 45,696
Percentage of Total Loans	1.82%	4.86%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 2,589	$ 0
Percentage of Total Loans	2.62%	0.00%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,364	$ 1,181
Percentage of Total Loans	0.87%	0.75%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 94,235	$ 90,300
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	48,583	37,100
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 142,818	$ 127,400
Percentage of Total Loans	15.65%	13.54%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 241	$ 184
Percentage of Total Loans	0.25%	0.18%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 2,794	$ 3,288
Percentage of Total Loans	1.79%	2.08%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 872	$ 417
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 872	$ 417
Percentage of Total Loans	0.10%	0.04%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 69,720	$ 82,103
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	866	1,095
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,436	767
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 72,022	$ 83,965
Percentage of Total Loans	99.57%	99.34%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 311	$ 557
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 311	$ 557
Percentage of Total Loans	0.43%	0.66%