Loans - Credit Quality Indicators by Class of Loans (Details) - USD ($) $ in Thousands	Sep. 30, 2018	Dec. 31, 2017	Sep. 30, 2017
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 962,743	$ 1,183,426	$ 1,235,969
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	295,802	435,207	447,482
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	90,444	90,287	90,088
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	394,416	454,051	473,046
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	136,151	146,751	155,676
Consumer and other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	45,338	56,398
Lease Financing Receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	592	732	$ 760
Pass
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	841,475	1,065,611
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	236,572	358,373
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	89,671	89,323
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	340,452	416,925
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	128,947	144,250
Pass | Consumer and other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	45,241	56,041
Pass | Lease Financing Receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	592	699
Special Mention
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	30,902	15,343
Special Mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	18,808	9,687
Special Mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	57	600
Special Mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	11,042	3,823
Special Mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	988	1,233
Special Mention | Consumer and other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	7	0
Special Mention | Lease Financing Receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Substandard
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	90,366	102,472
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	40,422	67,147
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	716	364
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	42,922	33,303
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	6,216	1,268
Substandard | Consumer and other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	90	357
Substandard | Lease Financing Receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	33
Doubtful
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Consumer and other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Lease Financing Receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 0