Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Financing Receivable, Recorded Investment [Line Items]
Total loans	$ 1,649,448	$ 1,627,817
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	27,479	19,681
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	5,619	9,479
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,522	7,182
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans		449
Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans	9,686	12,060
Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans	16,827	29,170
Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,632,621	1,598,647
Current | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	27,479	19,681
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans	36,075	39,587
SBA loans held for investment | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,558	792
SBA loans held for investment | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans		1,280
SBA loans held for investment | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans	510	2,473
SBA loans held for investment | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,068	4,545
SBA loans held for investment | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	34,007	35,042
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	46,450	118,257
SBA PPP loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	79
SBA PPP loans | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans	79
SBA PPP loans | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	46,371	118,257
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	931,726	839,788
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	704,674	630,423
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	89,670	71,404
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	109,903	118,280
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	334	3,109
Commercial loans | 30-59 days past due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans		1,047
Commercial loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans		186
Commercial loans | 60-89 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	565	1,971
Commercial loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	33	201
Commercial loans | Nonaccrual | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	366	1,059
Commercial loans | Nonaccrual | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,216	266
Commercial loans | Past Due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,265	6,139
Commercial loans | Past Due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans		1,047
Commercial loans | Past Due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	2,249	653
Commercial loans | Current | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans	703,409	624,284
Commercial loans | Current | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	89,670	70,357
Commercial loans | Current | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	107,654	117,627
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	409,355	467,586
Residential mortgage loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,688	3,232
Residential mortgage loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans		2,933
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans		262
Residential mortgage loans | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,262	5,217
Residential mortgage loans | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans	6,950	11,644
Residential mortgage loans | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	402,405	455,942
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	77,944	66,100
Consumer loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	120,525	87,164
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	76,725	62,549
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,219	3,551
Consumer loans | 30-59 days past due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans		120
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	39	393
Consumer loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans		3
Consumer loans | 60-89 days past due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	845	796
Consumer loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans		1
Consumer loans | 90 days and still accruing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans		187
Consumer loans | Nonaccrual | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,122	1,750
Consumer loans | Nonaccrual | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	210	1,295
Consumer loans | Past Due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	3,967	2,666
Consumer loans | Past Due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	249	1,875
Consumer loans | Past Due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans		4
Consumer loans | Current | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans	116,558	84,498
Consumer loans | Current | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans	76,476	60,674
Consumer loans | Current | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,219	3,547
Residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	120,525	87,164
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,014,251	997,632
Total
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,622,075	1,618,482
Total | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	5,619	8,882
Total | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,522	7,182
Total | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans		449
Total | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans	9,686	12,060
Total | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans	16,827	28,573
Total | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	1,605,248	1,589,909
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Total loans	27,373	9,335
SBA loans held for sale | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans		597
SBA loans held for sale | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans		597
SBA loans held for sale | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans	$ 27,373	$ 8,738