Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Financing Receivable, Recorded Investment [Line Items]
Total loans	$ 1,649,448	$ 1,627,817
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	27,479	19,681
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	27,479	19,681
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans	36,075	39,587
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans	34,959	36,818
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans	745	525
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans	371	2,244
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	46,450	118,257
SBA PPP loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans	46,450	118,257
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	931,726	839,788
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	704,674	630,423
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	89,670	71,404
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	109,903	118,280
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans	914,264	804,239
Commercial loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	694,627	603,482
Commercial loans | Pass | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	86,770	71,404
Commercial loans | Pass | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	105,388	109,672
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans	12,856	30,739
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	7,980	25,206
Commercial loans | Special mention | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,900
Commercial loans | Special mention | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,976	5,533
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans	4,606	4,810
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,067	1,735
Commercial loans | Substandard | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,539	3,075
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	409,355	467,586
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans	406,093	462,369
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,262	5,217
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	77,944	66,100
Consumer loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	120,525	87,164
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	76,725	62,549
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,219	3,551
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans	77,734	64,805
Consumer loans | Performing | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	117,403	85,414
Consumer loans | Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	76,515	61,254
Consumer loans | Performing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,219	3,551
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans	210	1,295
Consumer loans | Nonperforming | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,122	1,750
Consumer loans | Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	210	1,295
Residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	120,525	87,164
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	607,824	620,850
Total residential mortgage and consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans	601,230	612,588
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans	6,594	8,262
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,014,251	997,632
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans	995,673	959,314
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans	13,601	31,264
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans	4,977	7,054
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Total loans	$ 27,373	$ 9,335