Loans and Allowance for Loan and Lease Losses - Loan Balances (Details) $ in Thousands	12 Months Ended			24 Months Ended
	Dec. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)	Dec. 31, 2016 USD ($) loan	Dec. 31, 2013 USD ($)
Outstanding loan balances
Total gross loans held for investment	$ 1,385,462	$ 1,247,280		$ 1,385,462
Net deferred loan fees	(1,183)	(1,132)		(1,183)
Allowance for loan and lease losses	(17,237)	(17,452)	$ (16,802)	(17,237)	$ (17,859)
Net loans held for investment	1,367,042	1,228,696		1,367,042
Loans held for sale	10,055	9,755		10,055
Pledged Loans
Loans pledged as collateral to FHLB	748,800			748,800
FHLB secured credit facility	517,800			517,800
FHLB secured line of credit		10,200
Loans Serviced for Others
Unpaid principal balance of loans serviced for others, exclusive of SBA loans	33,300	38,000		33,300
Unpaid principal balance of SBA loans serviced for others	6,300	8,500		$ 6,300
Number of SBA loans sold | loan				0
Recognized gains from the sale of SBA loans			43
Available for sale Securities
Accretable yield, or income expected to be collected
Balance at the beginning of the period	3,821	4,374		$ 4,374
Additions			4,673
Accretion of income	(1,077)	(1,431)	(1,362)
Changes in expected cash flows	(104)	878	1,063
Balance at the end of the period	2,640	3,821	4,374	2,640
Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	1,377,570	1,238,505		1,377,570
Net deferred loan fees	(1,183)	(1,132)		(1,183)
Allowance for loan and lease losses	(17,237)	(17,373)		(17,237)
Net loans held for investment	1,359,150	1,220,000		1,359,150
Loans held for sale	10,055	9,755		10,055
PCI loans
Outstanding loan balances
Total gross loans held for investment	7,892	8,775		7,892
Allowance for loan and lease losses		(79)
Net loans held for investment	7,892	8,696		7,892
MISN
Outstanding loan balances
Total gross loans held for investment	140,300			140,300
MISN | Non-PCI Loans
Outstanding loan balances
Net loans held for investment	$ 132,400	$ 163,800		132,400
Minimum
Outstanding loan balances
Period for sale of loans categorized as held for sale	30 days
Maximum
Outstanding loan balances
Period for sale of loans categorized as held for sale		60 days
Residential 1 to 4 family
Concentration of Credit Risk
Loan portfolio collateralized by various forms of real estate	$ 1,100,000	$ 1,000,000		1,100,000
Real Estate Secured
Outstanding loan balances
Total gross loans held for investment	1,130,064	1,012,076		1,130,064
Real Estate Secured | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	1,124,114	1,005,536		1,124,114
Real Estate Secured | PCI loans
Outstanding loan balances
Total gross loans held for investment	5,950	6,540		5,950
Real Estate Secured | Commercial
Outstanding loan balances
Total gross loans held for investment	643,691	579,244		643,691
Real Estate Secured | Commercial | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	638,089	573,559		638,089
Real Estate Secured | Commercial | PCI loans
Outstanding loan balances
Total gross loans held for investment	5,602	5,685		5,602
Real Estate Secured | Residential 1 to 4 family
Outstanding loan balances
Total gross loans held for investment	205,999	165,829		205,999
Real Estate Secured | Residential 1 to 4 family | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	205,875	165,256		205,875
Real Estate Secured | Residential 1 to 4 family | PCI loans
Outstanding loan balances
Total gross loans held for investment	124	573		124
Real Estate Secured | Farmland
Outstanding loan balances
Total gross loans held for investment	155,771	120,566		155,771
Real Estate Secured | Farmland | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	155,771	120,566		155,771
Real Estate Secured | Multi-family residential
Outstanding loan balances
Total gross loans held for investment	77,941	79,381		77,941
Real Estate Secured | Multi-family residential | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	77,941	79,381		77,941
Real Estate Secured | Home equity lines of credit
Outstanding loan balances
Total gross loans held for investment	24,796	31,387		24,796
Real Estate Secured | Home equity lines of credit | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	24,796	31,387		24,796
Real Estate Secured | Construction and land
Outstanding loan balances
Total gross loans held for investment	21,866	35,669		21,866
Real Estate Secured | Construction and land | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	21,642	35,387		21,642
Real Estate Secured | Construction and land | PCI loans
Outstanding loan balances
Total gross loans held for investment	224	282		224
Real Estate Secured | Commercial and industrial | PCI loans
Outstanding loan balances
Total gross loans held for investment		783
Real Estate Secured | Agriculture | PCI loans
Outstanding loan balances
Total gross loans held for investment		1,452
Commercial
Outstanding loan balances
Total gross loans held for investment	251,202	229,171		251,202
Allowance for loan and lease losses	(5,249)	(5,372)	(5,125)	(5,249)	(4,781)
Commercial | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	249,260	226,936		249,260
Commercial | PCI loans
Outstanding loan balances
Total gross loans held for investment	1,942	2,235		1,942
Commercial | Commercial and industrial
Outstanding loan balances
Total gross loans held for investment	182,637	164,808		182,637
Commercial | Commercial and industrial | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	182,165	164,025		182,165
Commercial | Commercial and industrial | PCI loans
Outstanding loan balances
Total gross loans held for investment	472	783		472
Commercial | Agriculture
Outstanding loan balances
Total gross loans held for investment	68,565	64,363		68,565
Commercial | Agriculture | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	67,095	62,911		67,095
Commercial | Agriculture | PCI loans
Outstanding loan balances
Total gross loans held for investment	1,470	1,452		1,470
Consumer
Outstanding loan balances
Total gross loans held for investment	4,196	6,033		4,196
Allowance for loan and lease losses	(111)	(173)	$ (202)	(111)	$ (131)
Consumer | Non-PCI Loans
Outstanding loan balances
Total gross loans held for investment	$ 4,196	$ 6,033		$ 4,196