Loans - Classification of Loans By Class (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 1,581,905	$ 1,412,029
SBA loans held for sale	10,602	13,529
Total loans	1,592,507	1,425,558
SBA 504 loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	23,299	26,726
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	469,987	467,706
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	146,161	143,524
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	36,966	35,767
SBA PPP loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	136,039	0
Commercial loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	792,752	765,032
Commercial loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	121,961	112,014
Commercial loans | Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	586,332	578,643
Commercial loans | Commercial real estate construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	61,160	47,649
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	469,987	467,706
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	146,161	143,524
Consumer loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	78,569	73,935
Consumer loans | Home equity
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 67,592	$ 69,589