Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	$ 14,106	$ 17,517
Loans receivable, nonaccrual status	9,473	5,649
Loan receivable, current	1,578,401	1,408,041
Total loans held for investment	1,567,799	1,394,512
Total loans	1,581,905	1,412,029
SBA loans held for sale	10,602	13,529
Total loans	1,592,507	1,425,558
SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		1,808
Loan receivable, current	23,299	24,918
Total loans	23,299	26,726
30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	3,956	7,163
60-89 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	677	3,775
60-89 days past due | SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		1,808
90 days and still accruing
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		930
SBA loans held for investment
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	3,765	2,212
Loans receivable, nonaccrual status	2,331	1,164
Loan receivable, current	33,201	33,555
Total loans	36,966	35,767
SBA loans held for investment | 30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	828	1,048
SBA loans held for investment | 60-89 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	606
SBA PPP loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	32
Loans receivable, nonaccrual status	32
Loan receivable, current	136,007
Total loans	136,039	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans	792,752	765,032
Commercial loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	81	387
Loans receivable, nonaccrual status	10	316
Loan receivable, current	121,880	111,627
Total loans	121,961	112,014
Commercial loans | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	403	428
Loans receivable, nonaccrual status	403	213
Loan receivable, current	585,929	578,215
Total loans	586,332	578,643
Commercial loans | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Loan receivable, current	61,160	47,649
Total loans	61,160	47,649
Commercial loans | 30-59 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		71
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		215
Commercial loans | 60-89 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	71
Residential mortgage loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	9,150	10,925
Loans receivable, nonaccrual status	6,192	3,936
Loan receivable, current	460,837	456,781
Total loans	469,987	467,706
Residential mortgage loans | 30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	2,958	4,383
Residential mortgage loans | 60-89 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		1,676
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		930
Consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans	146,161	143,524
Consumer loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		133
Loans receivable, nonaccrual status		20
Loan receivable, current	78,569	73,802
Total loans	78,569	73,935
Consumer loans | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	675	1,624
Loans receivable, nonaccrual status	505
Loan receivable, current	66,917	67,965
Total loans	67,592	69,589
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	170	1,446
Consumer loans | 60-89 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		113
Consumer loans | 60-89 days past due | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		178
Collateral Pledged | Small Business Administration
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, nonaccrual status	$ 307	$ 59