Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 993,228	$ 979,449
Less: Unearned origination fees, net of origination costs	(3,975)	(4,434)
Total loans	989,253	975,015
AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	939,684	939,573
AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	16,880	2,586
AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	36,664	37,290
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	315,512	278,178
Commercial | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	285,057	257,639
Commercial | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,141	1,950
Commercial | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,314	18,589
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,543	26,061
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,543	26,061
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	83,884	72,159
Real estate construction other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	83,884	72,159
Real estate construction other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	135,053	152,178
Real estate term owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	125,600	135,359
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	2,124	57
Real estate term owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	7,329	16,762
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	344,968	356,601
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	333,382	355,903
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	11,004	0
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	582	698
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	41,326	45,402
Real estate term other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	40,688	44,733
Real estate term other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	638	669
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	23,333	23,589
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	22,181	22,568
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	380	501
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	772	520
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	21,609	25,281
Consumer other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	21,349	25,151
Consumer other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	231	78
Consumer other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 29	$ 52