Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Mar. 31, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 986,641	$ 988,833
Less: Unearned origination fees, net of origination costs	(4,300)	(4,487)
Total loans	982,341	984,346
AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	932,997	928,552
AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	25,316	31,426
AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	28,327	28,854
AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1	1
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	344,164	342,420
Total loans	342,857	341,091
Commercial | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	323,263	315,112
Commercial | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,467	5,116
Commercial | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	17,434	22,192
Commercial | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	36,484	37,111
Total loans	36,286	36,828
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	34,061	33,729
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	3,382
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	2,423	0
Real estate construction one-to-four family | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	72,920	72,256
Total loans	72,421	71,658
Real estate construction other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	72,920	72,256
Real estate construction other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	130,141	126,414
Total loans	129,491	125,795
Real estate term owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	120,469	117,174
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,886	3,987
Real estate term owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	5,786	5,253
Real estate term owner occupied | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	316,161	325,720
Total loans	314,663	324,198
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	297,268	307,126
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	17,963	18,129
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	930	465
Real estate term non-owner occupied | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	43,910	42,039
Total loans	43,598	41,746
Real estate term other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	42,682	40,792
Real estate term other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	670
Real estate term other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,228	577
Real estate term other | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	18,716	19,228
Total loans	18,721	19,234
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	18,259	18,768
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	140
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	457	320
Consumer secured by 1st deeds of trust | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,145	23,645
Total loans	24,304	23,796
Consumer other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,075	23,595
Consumer other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	2
Consumer other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	69	47
Consumer other | AQR Doubtful
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 1	$ 1