Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	$ 23,105	$ 16,395
Charge-offs	(656)	(525)
Recoveries	16	6
Net (charge-offs) recoveries	(640)	(519)
Provision for (credit to) loan losses charged to expense	500	1,500
Balance, end of period	22,965	17,376
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	1,301	1,079
Charge-offs	(282)	(25)
Recoveries	15	5
Net (charge-offs) recoveries	(267)	(20)
Provision for (credit to) loan losses charged to expense	620	(54)
Balance, end of period	1,654	1,005
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	14,992	9,722
Charge-offs	(373)	(300)
Recoveries	1	1
Net (charge-offs) recoveries	(372)	(299)
Provision for (credit to) loan losses charged to expense	107	706
Balance, end of period	14,727	10,129
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	5,318	4,254
Charge-offs		(200)
Net (charge-offs) recoveries		(200)
Provision for (credit to) loan losses charged to expense	(309)	709
Balance, end of period	5,009	4,763
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	681	625
Charge-offs	(1)
Net (charge-offs) recoveries	(1)
Provision for (credit to) loan losses charged to expense	(131)	46
Balance, end of period	549	671
Consumer construction loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	813	715
Provision for (credit to) loan losses charged to expense	213	93
Balance, end of period	$ 1,026	$ 808