Loans and Allowance for Credit Losses on Loans - Loans Balance (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019	Jun. 30, 2020	Jan. 01, 2020	Dec. 31, 2019	Jun. 30, 2019	Dec. 31, 2018
Loans held-for-investment:
Total loan balance	$ 2,705,479		$ 2,705,479				$ 2,534,163
Deferred loan fees, net	(8,463)		(8,463)				(319)
Loans, net of deferred fees	2,697,016	$ 1,875,387	2,697,016	$ 1,875,387			2,533,844
Allowance for credit losses on loans(1)	(45,422)	(25,895)	(45,422)	(25,895)	$ (45,444)	$ (31,855)	(23,285)	$ (26,631)	$ (27,848)
Loans, net	2,651,594		2,651,594				2,510,559
Income tax expense	4,198	4,633	9,340	13,763
Retained earnings	91,065		91,065				96,741
Commercial
Loans held-for-investment:
Total loan balance							603,345
Allowance for credit losses on loans(1)		(14,656)		(14,656)			(10,453)	(15,234)	(17,061)
Commercial | SBA Paycheck Protection Program ("PPP")
Loans held-for-investment:
Loans, net	323,550		323,550
Real estate
Loans held-for-investment:
Total loan balance							1,897,074
Allowance for credit losses on loans(1)		(11,143)		(11,143)			(12,750)	(11,307)	(10,671)
Consumer
Loans held-for-investment:
Total loan balance							33,744
Allowance for credit losses on loans(1)		$ (96)		$ (96)			(82)	$ (90)	$ (116)
Commercial
Loans held-for-investment:
Total loan balance	897,909		897,909			6,790
Allowance for credit losses on loans(1)	(12,800)		(12,800)		(13,179)	(6,790)	(10,453)
Commercial | Commercial
Loans held-for-investment:
Total loan balance	897,909		897,909				603,345
SBA Paycheck Protection Program ("PPP") loans | Commercial
Loans held-for-investment:
Total loan balance	323,550		323,550
CRE - owner occupied
Loans held-for-investment:
Total loan balance	561,528		561,528			6,994
Allowance for credit losses on loans(1)	(9,283)		(9,283)		(8,547)	(6,994)	(3,825)
CRE - owner occupied | Real estate
Loans held-for-investment:
Total loan balance	561,528		561,528				548,907
CRE - non-owner occupied
Loans held-for-investment:
Total loan balance	713,563		713,563			11,672	767,821
Allowance for credit losses on loans(1)	(15,325)		(15,325)		(15,449)	(11,672)	(3,760)
CRE - non-owner occupied | Real estate
Loans held-for-investment:
Total loan balance	713,563		713,563				767,821
Land and construction
Loans held-for-investment:
Total loan balance	142,632		142,632			1,458
Allowance for credit losses on loans(1)	(2,544)		(2,544)		(2,552)	(1,458)	(2,621)
Land and construction | Real estate
Loans held-for-investment:
Total loan balance	142,632		142,632				147,189
Home equity
Loans held-for-investment:
Total loan balance	111,468		111,468			1,321
Allowance for credit losses on loans(1)	(1,881)		(1,881)		(1,851)	(1,321)	(2,244)
Home equity | Real estate
Loans held-for-investment:
Total loan balance	111,468		111,468				151,775
Multi-family
Loans held-for-investment:
Total loan balance	169,791		169,791			1,253
Allowance for credit losses on loans(1)	(1,849)		(1,849)		(1,828)	(1,253)	(57)
Multi-family | Real estate
Loans held-for-investment:
Total loan balance	169,791		169,791				180,623
Residential mortgages
Loans held-for-investment:
Total loan balance	91,077		91,077			678
Allowance for credit losses on loans(1)	(779)		(779)		(825)	(678)	(243)
Residential mortgages | Real estate
Loans held-for-investment:
Total loan balance	91,077		91,077				100,759
Consumer and other
Loans held-for-investment:
Total loan balance	17,511		17,511			1,689
Allowance for credit losses on loans(1)	(961)		(961)		$ (1,213)	$ (1,689)	(82)
Consumer and other | Consumer
Loans held-for-investment:
Total loan balance	$ 17,511		$ 17,511				33,744
ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans(1)							(8,570)
ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Allowance for credit losses on loans(1)							3,663
ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Allowance for credit losses on loans(1)							(3,169)
ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Allowance for credit losses on loans(1)							(7,912)
ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Allowance for credit losses on loans(1)							1,163
ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Allowance for credit losses on loans(1)							923
ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Allowance for credit losses on loans(1)							(1,196)
ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Allowance for credit losses on loans(1)							(435)
ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Allowance for credit losses on loans(1)							(1,607)
Restatement Adjustment | ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans(1)							(8,570)
Retained earnings							(6,062)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Total loan balance							(3,663)
Restatement Adjustment | ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Total loan balance							3,169
Restatement Adjustment | ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Total loan balance							7,912
Restatement Adjustment | ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Total loan balance							(1,163)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Total loan balance							(923)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Total loan balance							1,196
Restatement Adjustment | ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Total loan balance							435
Restatement Adjustment | ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Total loan balance							1,607
Previously Reported | ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans(1)							(23,285)
Previously Reported | ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Total loan balance							10,453
Previously Reported | ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Total loan balance							3,825
Previously Reported | ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Total loan balance							3,760
Previously Reported | ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Total loan balance							2,621
Previously Reported | ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Total loan balance							2,244
Previously Reported | ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Total loan balance							57
Previously Reported | ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Total loan balance							243
Previously Reported | ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Total loan balance							$ 82