Credit Quality of Loans and Allowance for Loan Losses - Credit Quality Indicators by Class of Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017	Jun. 30, 2017
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,057,963	$ 1,183,426	$ 1,240,253
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	354,944	435,207	451,767
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	414,526	448,406	461,064
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 769,470	$ 883,613
Percentage of total loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 98,108	$ 90,287	98,695
Percentage of total loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 141,104	$ 146,751	156,394
Percentage of total loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 47,406	$ 56,398	70,031
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	632	732	866
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,243	5,645	$ 1,436
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 49,281	$ 62,775
Percentage of total loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 275,744	$ 358,373
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	361,281	411,280
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 637,025	$ 769,653
Percentage of total loans	82.79%	87.10%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 96,225	$ 89,323
Percentage of total loans	98.08%	98.93%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 134,952	$ 144,250
Percentage of total loans	95.64%	98.30%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 21,320	$ 9,687
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	9,985	3,823
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 31,305	$ 13,510
Percentage of total loans	4.07%	1.53%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 85	$ 600
Percentage of total loans	0.09%	0.67%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 602	$ 1,233
Percentage of total loans	0.43%	0.84%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 57,880	$ 67,147
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	43,260	33,303
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 101,140	$ 100,450
Percentage of total loans	13.14%	11.37%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,798	$ 364
Percentage of total loans	1.83%	0.40%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,550	$ 1,268
Percentage of total loans	3.93%	0.86%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 47,355	$ 56,041
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	632	699
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,243	5,645
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 49,230	$ 62,385
Percentage of total loans	99.90%	99.38%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 51	$ 357
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	33
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 51	$ 390
Percentage of total loans	0.10%	0.62%