Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	$ 17,376	$ 15,684	$ 16,395	$ 15,488
Charge-offs	(219)	(84)	(744)	(395)
Recoveries	577	15	583	22
Net (charge-offs) recoveries	358	(69)	(161)	(373)
Provision for loan losses charged to expense	2,500	350	4,000	850
Ending Balance	20,234	15,965	20,234	15,965
SBA loans held for investment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,005	1,678	1,079	1,655
Charge-offs		(84)	(25)	(393)
Recoveries	75	1	80	3
Net (charge-offs) recoveries	75	(83)	55	(390)
Provision for loan losses charged to expense	(76)	(274)	(130)	56
Ending Balance	1,004	1,321	1,004	1,321
Commercial loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	10,129	8,793	9,722	8,705
Charge-offs	(219)		(519)	(1)
Recoveries	502	4	503	9
Net (charge-offs) recoveries	283	4	(16)	8
Provision for loan losses charged to expense	1,924	347	2,630	431
Ending Balance	12,336	9,144	12,336	9,144
Residential mortgage loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	4,763	4,016	4,254	3,900
Charge-offs			(200)
Net (charge-offs) recoveries			(200)
Provision for loan losses charged to expense	676	182	1,385	298
Ending Balance	5,439	4,198	5,439	4,198
Consumer loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,479	1,197	1,340	1,228
Charge-offs				(1)
Recoveries		10		10
Net (charge-offs) recoveries		10		9
Provision for loan losses charged to expense	(24)	95	115	65
Ending Balance	$ 1,455	$ 1,302	$ 1,455	$ 1,302