Loans and Allowance for Credit Losses on Loans - Loans Balance (Details) - USD ($)	Dec. 31, 2020	Jan. 01, 2020	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Loans held-for-investment:
Total loan balance	$ 2,625,987,000	$ 31,855,000	$ 2,534,163,000
Deferred loan fees, net	(6,726,000)		(319,000)
Loans, net of deferred fees	2,619,261,000		2,533,844,000	$ 1,886,405,000
Allowance for credit losses on loans(1)	(44,400,000)	(31,855,000)	(23,285,000)	(27,848,000)	$ (19,658,000)
Loans, net	2,574,861,000		2,510,559,000
Commercial
Loans held-for-investment:
Total loan balance			603,345,000
Allowance for credit losses on loans(1)			(10,453,000)	(17,061,000)	(10,608,000)
Real estate
Loans held-for-investment:
Total loan balance			1,897,074,000
Allowance for credit losses on loans(1)			(12,750,000)	(10,671,000)	(8,950,000)
Consumer
Loans held-for-investment:
Total loan balance			33,744,000
Allowance for credit losses on loans(1)			(82,000)	$ (116,000)	$ (100,000)
Commercial
Loans held-for-investment:
Total loan balance	846,386,000	6,790,000	603,345,000
Allowance for credit losses on loans(1)	(11,587,000)	(6,790,000)	(10,453,000)
Commercial | Commercial
Loans held-for-investment:
Total loan balance	846,386,000		603,345,000
SBA Paycheck Protection Program ("PPP") loans | Commercial
Loans held-for-investment:
Total loan balance	290,679,000
CRE - owner occupied
Loans held-for-investment:
Total loan balance	560,362,000	6,994,000	548,907,000
Allowance for credit losses on loans(1)	(8,560,000)	(6,994,000)	(3,825,000)
CRE - owner occupied | Real estate
Loans held-for-investment:
Total loan balance	560,362,000		548,907,000
CRE - non-owner occupied
Loans held-for-investment:
Total loan balance	693,103,000	11,672,000	767,821,000
Allowance for credit losses on loans(1)	(16,416,000)	(11,672,000)	(3,760,000)
CRE - non-owner occupied | Real estate
Loans held-for-investment:
Total loan balance	693,103,000		767,821,000
Land and construction
Loans held-for-investment:
Total loan balance	144,594,000	1,458,000	147,189,000
Allowance for credit losses on loans(1)	(2,509,000)	(1,458,000)	(2,621,000)
Land and construction | Real estate
Loans held-for-investment:
Total loan balance	144,594,000		147,189,000
Home equity
Loans held-for-investment:
Total loan balance	111,885,000	1,321,000	151,775,000
Allowance for credit losses on loans(1)	(1,297,000)	(1,321,000)	(2,244,000)
Home equity | Real estate
Loans held-for-investment:
Total loan balance	111,885,000		151,775,000
Multi-family
Loans held-for-investment:
Total loan balance	166,425,000	1,253,000	180,623,000
Allowance for credit losses on loans(1)	(2,804,000)	(1,253,000)	(57,000)
Multi-family | Real estate
Loans held-for-investment:
Total loan balance	166,425,000		180,623,000
Residential mortgages
Loans held-for-investment:
Total loan balance	85,116,000	678,000	100,759,000
Allowance for credit losses on loans(1)	(943,000)	(678,000)	(243,000)
Residential mortgages | Real estate
Loans held-for-investment:
Total loan balance	85,116,000		100,759,000
Consumer and other
Loans held-for-investment:
Total loan balance	18,116,000	1,689,000	33,744,000
Allowance for credit losses on loans(1)	(284,000)	$ (1,689,000)	(82,000)
Consumer and other | Consumer
Loans held-for-investment:
Total loan balance	$ 18,116,000		33,744,000
ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans(1)			8,570,000
ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Allowance for credit losses on loans(1)			(3,663,000)
ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Allowance for credit losses on loans(1)			3,169,000
ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Allowance for credit losses on loans(1)			7,912,000
ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Allowance for credit losses on loans(1)			(1,163,000)
ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Allowance for credit losses on loans(1)			(923,000)
ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Allowance for credit losses on loans(1)			1,196,000
ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Allowance for credit losses on loans(1)			435,000
ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Allowance for credit losses on loans(1)			1,607,000
Restatement Adjustment | ASU 2016-03 - Topic 326
Loans held-for-investment:
Total loan balance			8,570,000
Allowance for credit losses on loans(1)			(8,570,000)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Total loan balance			(3,663,000)
Restatement Adjustment | ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Total loan balance			3,169,000
Restatement Adjustment | ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Total loan balance			7,912,000
Restatement Adjustment | ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Total loan balance			(1,163,000)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Total loan balance			(923,000)
Restatement Adjustment | ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Total loan balance			1,196,000
Restatement Adjustment | ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Total loan balance			435,000
Restatement Adjustment | ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Total loan balance			1,607,000
Previously Reported | ASU 2016-03 - Topic 326
Loans held-for-investment:
Total loan balance			23,285,000
Previously Reported | ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Total loan balance			10,453,000
Previously Reported | ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Total loan balance			3,825,000
Previously Reported | ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Total loan balance			3,760,000
Previously Reported | ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Total loan balance			2,621,000
Previously Reported | ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Total loan balance			2,244,000
Previously Reported | ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Total loan balance			57,000
Previously Reported | ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Total loan balance			243,000
Previously Reported | ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Total loan balance			$ 82,000