Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015	Mar. 31, 2015
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,250,049	$ 1,263,645	$ 1,310,929
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	441,160	454,028	484,508
Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	64,820	56,898
Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	467,648	471,141
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 973,628	$ 982,067
Percentage of Total Loans	100.00%	100.00%
Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 19,970	$ 18,054
Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	149,961	149,064
Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 169,931	$ 167,118
Percentage of Total Loans	100.00%	100.00%
Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,718	$ 6,055
Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	97,463	104,954
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,590	1,968	6,350
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,719	1,483	$ 2,439
Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 106,490	$ 114,460
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 368,557	$ 383,897
Pass | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	64,602	56,740
Pass | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	406,237	412,141
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 839,396	$ 852,778
Percentage of Total Loans	86.22%	86.84%
Pass | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 19,970	$ 18,054
Pass | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	145,302	144,704
Pass | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 165,272	$ 162,758
Percentage of Total Loans	97.26%	97.39%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 24,933	$ 32,506
Special mention | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	99	34
Special mention | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	24,759	28,217
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 49,791	$ 60,757
Percentage of Total Loans	5.11%	6.18%
Special mention | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 0
Special mention | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,122	1,225
Special mention | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,122	$ 1,225
Percentage of Total Loans	0.66%	0.73%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 47,466	$ 37,353
Substandard | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	119	124
Substandard | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	36,652	30,783
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 84,237	$ 68,260
Percentage of Total Loans	8.65%	6.95%
Substandard | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 0
Substandard | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	3,537	3,135
Substandard | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,537	$ 3,135
Percentage of Total Loans	2.08%	1.88%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 204	$ 272
Doubtful | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 204	$ 272
Percentage of Total Loans	0.02%	0.03%
Performing | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,702	$ 6,033
Performing | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	96,919	104,503
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,590	1,968
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,719	1,483
Performing | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 105,930	$ 113,987
Percentage of Total Loans	99.47%	99.59%
Nonperforming | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 16	$ 22
Nonperforming | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	544	451
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 560	$ 473
Percentage of Total Loans	0.53%	0.41%