Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended										12 Months Ended
	Dec. 31, 2022	Sep. 30, 2022	Jun. 30, 2022	Mar. 31, 2022	Dec. 31, 2021	Mar. 31, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				$ 22,302		$ 23,105				$ 16,395	$ 22,302	$ 23,105	$ 16,395
Charge-offs											(1,457)	(1,146)	(895)
Recoveries											192	162	605
Net recoveries (charge-offs)											(1,265)	(984)	(290)
Provision for (credit to) loan losses charged to expense	$ 1,632	$ 1,517	$ 1,188	(178)	$ (319)	500	$ 1,000	$ 2,000	$ 2,500	1,500	4,159	181	7,000
Balance, end of period	25,196				22,302		23,105				25,196	22,302	23,105
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				1,074		1,301				1,079	1,074	1,301	1,079
Charge-offs											(59)	(591)	(26)
Recoveries											33	86	83
Net recoveries (charge-offs)											(26)	(505)	57
Provision for (credit to) loan losses charged to expense											(173)	278	165
Balance, end of period	875				1,074		1,301				875	1,074	1,301
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				15,053		14,992				9,722	15,053	14,992	9,722
Charge-offs											(1,000)	(551)	(669)
Recoveries											109	34	522
Net recoveries (charge-offs)											(891)	(517)	(147)
Provision for (credit to) loan losses charged to expense											1,092	578	5,417
Balance, end of period	15,254				15,053		14,992				15,254	15,053	14,992
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				4,114		5,318				4,254	4,114	5,318	4,254
Charge-offs											0	0	(200)
Recoveries											3	42
Net recoveries (charge-offs)											3	42	(200)
Provision for (credit to) loan losses charged to expense											1,333	(1,246)	1,264
Balance, end of period	5,450				4,114		5,318				5,450	4,114	5,318
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				671		681				625	671	681	625
Charge-offs											(398)	(4)
Recoveries											47
Net recoveries (charge-offs)											(351)	(4)
Provision for (credit to) loan losses charged to expense											670	(6)	56
Balance, end of period	990				671		681				990	671	681
Residential construction loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period				$ 1,390		$ 813				$ 715	1,390	813	715
Provision for (credit to) loan losses charged to expense											1,237	577	98
Balance, end of period	$ 2,627				$ 1,390		$ 813				$ 2,627	$ 1,390	$ 813