Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended										12 Months Ended
	Dec. 31, 2021	Mar. 31, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		$ 23,105				$ 16,395				$ 15,488	$ 23,105	$ 16,395	$ 15,488
Charge-offs											(1,146)	(895)	(1,242)
Recoveries											162	605	49
Net (charge-offs) recoveries											(984)	(290)	(1,193)
Provision for (credit to) loan losses charged to expense	$ (319)	500	$ 1,000	$ 2,000	$ 2,500	1,500	$ 500	$ 750	$ 350	500	181	7,000	2,100
Balance, end of period	22,302		23,105				16,395				22,302	23,105	16,395
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		1,301				1,079				1,655	1,301	1,079	1,655
Charge-offs											(591)	(26)	(535)
Recoveries											86	83	23
Net (charge-offs) recoveries											(505)	57	(512)
Provision for (credit to) loan losses charged to expense											278	165	(64)
Balance, end of period	1,074		1,301				1,079				1,074	1,301	1,079
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		14,992				9,722				8,705	14,992	9,722	8,705
Charge-offs											(551)	(669)	(501)
Recoveries											34	522	16
Net (charge-offs) recoveries											(517)	(147)	(485)
Provision for (credit to) loan losses charged to expense											578	5,417	1,502
Balance, end of period	15,053		14,992				9,722				15,053	14,992	9,722
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		5,318				4,254				3,900	5,318	4,254	3,900
Charge-offs											0	(200)	(205)
Recoveries											42	0
Net (charge-offs) recoveries											42	(200)	(205)
Provision for (credit to) loan losses charged to expense											(1,246)	1,264	559
Balance, end of period	4,114		5,318				4,254				4,114	5,318	4,254
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		681				625				618	681	625	618
Charge-offs											(4)		(1)
Recoveries													10
Net (charge-offs) recoveries											(4)		9
Provision for (credit to) loan losses charged to expense											(6)	56	(2)
Balance, end of period	671		681				625				671	681	625
Residential construction loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period		$ 813				$ 715				$ 610	813	715	610
Provision for (credit to) loan losses charged to expense											577	98	105
Balance, end of period	$ 1,390		$ 813				$ 715				$ 1,390	$ 813	$ 715