Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Jun. 30, 2015	Dec. 31, 2014	Jun. 30, 2014
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,294,392	$ 1,284,431	$ 1,224,182
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	471,397	467,147	454,310
Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	57,528	43,510
Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	469,022	467,172
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 997,947	$ 977,829
Percentage of Total Loans	100.00%	100.00%
Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 21,648	$ 25,067
Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	153,820	154,602
Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 175,468	$ 179,669
Percentage of Total Loans	100.00%	100.00%
Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,817	$ 6,012
Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	107,809	113,316
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,561	4,857	4,750
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,790	2,748	$ 3,656
Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 120,977	$ 126,933
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 418,761	$ 456,221
Pass | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	57,361	43,320
Pass | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	421,359	440,281
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 897,481	$ 939,822
Percentage of Total Loans	89.93%	96.11%
Pass | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 20,666	$ 25,067
Pass | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	149,209	150,664
Pass | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 169,875	$ 175,731
Percentage of Total Loans	96.81%	97.81%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 19,477	$ 4,861
Special mention | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	126	132
Special mention | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	14,039	7,120
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 33,642	$ 12,113
Percentage of Total Loans	3.37%	1.24%
Special mention | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 492	$ 0
Special mention | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,320	1,184
Special mention | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,812	$ 1,184
Percentage of Total Loans	1.03%	0.66%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 32,812	$ 5,541
Substandard | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	41	58
Substandard | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	33,624	19,771
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 66,477	$ 25,370
Percentage of Total Loans	6.66%	2.60%
Substandard | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 490	$ 0
Substandard | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	3,291	2,754
Substandard | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,781	$ 2,754
Percentage of Total Loans	2.16%	1.53%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 347	$ 524
Doubtful | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 347	$ 524
Percentage of Total Loans	0.04%	0.05%
Performing | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,817	$ 5,995
Performing | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	107,809	112,893
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,561	4,857
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,790	2,748
Performing | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 120,977	$ 126,493
Percentage of Total Loans	100.00%	99.65%
Nonperforming | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 17
Nonperforming | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	423
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 440
Percentage of Total Loans	0.00%	0.35%