Loans and Asset Quality - Total Loans Held for Investment by Category and Loans Held for Sale (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	$ 1,602,086	$ 1,588,446
Total loans HFS	18,449	29,116
Real estate | Commercial real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	562,616	556,769
Real estate | One-to-four family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	447,420	442,889
Real estate | Construction and development
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	117,952	127,321
Commercial and industrial
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	266,180	250,428
SBA PPP, net of deferred income
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	119,358	118,447
Tax-exempt
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	66,554	68,666
Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	$ 22,006	$ 23,926