Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Dec. 31, 2020
Financing Receivable, Recorded Investment [Line Items]
Loans	$ 1,642,674	$ 1,618,482
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans	22,155	19,681
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans	22,155	19,681
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Loans	38,114	39,587
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans	36,714	36,991
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Loans	524	525
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans	876	2,071
SBA and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Loans	1,056,055	997,632
SBA and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans	1,016,008	959,487
SBA and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Loans	30,391	31,264
SBA and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans	9,656	6,881
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Loans	132,375	118,257
SBA PPP loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans	132,375	118,257
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Loans	885,566	839,788
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans	676,383	630,423
Commercial loans | Construction
Financing Receivable, Recorded Investment [Line Items]
Loans	73,250	71,404
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	113,778	118,280
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans	846,919	804,239
Commercial loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans	648,520	603,482
Commercial loans | Pass | Construction
Financing Receivable, Recorded Investment [Line Items]
Loans	73,250	71,404
Commercial loans | Pass | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	102,994	109,672
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Loans	29,867	30,739
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans	22,285	25,206
Commercial loans | Special mention | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	7,582	5,533
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans	8,780	4,810
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans	5,578	1,735
Commercial loans | Substandard | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	3,202	3,075
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Loans	422,188	467,586
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Loans	418,145	462,369
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Loans	4,043	5,217
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Loans	64,557	66,100
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans	63,390	62,549
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	1,167	3,551
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Loans	64,554	64,805
Consumer loans | Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans	63,390	61,254
Consumer loans | Performing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	1,164	3,551
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Loans	3	1,295
Consumer loans | Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans		1,295
Consumer loans | Nonperforming | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans	3
Residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Loans	99,874	87,164
Residential construction loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Loans	97,585	85,414
Residential construction loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Loans	2,289	1,750
Residential mortgage, consumer and residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Loans	586,619	620,850
Residential mortgage, consumer and residential construction loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Loans	580,284	612,588
Residential mortgage, consumer and residential construction loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Loans	$ 6,335	$ 8,262