Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015	Jun. 30, 2015
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,262,389	$ 1,263,645	$ 1,294,392
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	456,264	454,028	471,397
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	463,142	471,141	469,022
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 919,406	$ 925,169
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 96,331	$ 74,952	79,176
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 148,379	$ 149,064
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 94,522	$ 111,009	113,626
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,641	1,968	5,561
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	2,110	1,483	$ 1,790
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 98,273	$ 114,460
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 372,933	$ 383,897
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	399,866	412,141
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 772,799	$ 796,038
Percentage of Total Loans	84.06%	86.04%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 96,117	$ 74,794
Percentage of Total Loans	99.78%	99.79%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 144,523	$ 144,704
Percentage of Total Loans	97.40%	97.08%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 32,211	$ 32,506
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	26,353	28,217
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 58,564	$ 60,723
Percentage of Total Loans	6.37%	6.55%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 34
Percentage of Total Loans	0.00%	0.04%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 202	$ 1,225
Percentage of Total Loans	0.14%	0.82%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 50,896	$ 37,353
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	36,923	30,783
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 87,819	$ 68,136
Percentage of Total Loans	9.55%	7.36%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 214	$ 124
Percentage of Total Loans	0.22%	0.17%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,654	$ 3,135
Percentage of Total Loans	2.46%	2.10%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 224	$ 272
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 224	$ 272
Percentage of Total Loans	0.02%	0.03%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 94,029	$ 110,536
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,641	1,968
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	2,110	1,483
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 97,780	$ 113,987
Percentage of Total Loans	99.50%	99.59%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 493	$ 473
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 493	$ 473
Percentage of Total Loans	0.50%	0.41%