Credit Quality of Loans and Allowance for Loan Losses - Age Analysis of Past Due Loans (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	$ 20,277	$ 6,399
60-89 Days Past Due	2,345	3,446
Greater than 90 Days Past Due	42,521	47,192
Total Past Due	65,143	57,037
Current	1,184,906	1,206,608
Total Loans	1,250,049	1,263,645
Recorded Investment, 90 days past due and Accruing	258	147
Commercial, financial and agricultural
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	6,021	1,362
60-89 Days Past Due	1,922	2,317
Greater than 90 Days Past Due	24,116	25,696
Total Past Due	32,059	29,375
Current	409,101	424,653
Total Loans	441,160	454,028
Recorded Investment, 90 days past due and Accruing	204	59
Commercial real estate - construction
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	260	1,047
60-89 Days Past Due	0	0
Greater than 90 Days Past Due	11	12
Total Past Due	271	1,059
Current	64,549	55,839
Total Loans	64,820	56,898
Recorded Investment, 90 days past due and Accruing	0	0
Commercial real estate - other
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	10,754	1,164
60-89 Days Past Due	0	514
Greater than 90 Days Past Due	16,275	19,512
Total Past Due	27,029	21,190
Current	440,619	449,951
Total Loans	467,648	471,141
Recorded Investment, 90 days past due and Accruing	0	0
Residential - construction
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	1,468	0
60-89 Days Past Due	0	0
Greater than 90 Days Past Due	0	0
Total Past Due	1,468	0
Current	18,502	18,054
Total Loans	19,970	18,054
Recorded Investment, 90 days past due and Accruing	0	0
Residential - prime
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	1,046	1,703
60-89 Days Past Due	97	367
Greater than 90 Days Past Due	1,625	1,563
Total Past Due	2,768	3,633
Current	147,193	145,431
Total Loans	149,961	149,064
Recorded Investment, 90 days past due and Accruing	0	19
Consumer - credit card
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	37	38
60-89 Days Past Due	17	25
Greater than 90 Days Past Due	16	22
Total Past Due	70	85
Current	5,648	5,970
Total Loans	5,718	6,055
Recorded Investment, 90 days past due and Accruing	16	22
Consumer - other
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	625	984
60-89 Days Past Due	306	219
Greater than 90 Days Past Due	478	387
Total Past Due	1,409	1,590
Current	96,054	103,364
Total Loans	97,463	104,954
Recorded Investment, 90 days past due and Accruing	38	47
Lease financing receivable
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	0	0
60-89 Days Past Due	0	0
Greater than 90 Days Past Due	0	0
Total Past Due	0	0
Current	1,590	1,968
Total Loans	1,590	1,968
Recorded Investment, 90 days past due and Accruing	0	0
Other loans
Age Analysis of Past Due Loans by Class of Loans [Abstract]
30-59 Days Past Due	66	101
60-89 Days Past Due	3	4
Greater than 90 Days Past Due	0	0
Total Past Due	69	105
Current	1,650	1,378
Total Loans	1,719	1,483
Recorded Investment, 90 days past due and Accruing	$ 0	$ 0