Loans - Loan Portfolio by Internal Risk Rating (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	$ 13,236,433	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	8,225,765	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,675,085	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	5,171,356	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	321,993	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	57,331	68,431
Investor loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		4,147,989
Investor loans secured by real estate | Pass | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,617,655	2,067,875
Investor loans secured by real estate | Pass | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	5,156,988	1,575,510
Investor loans secured by real estate | Pass | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	321,993	438,769
Investor loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	44,754	65,835
Investor loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		2,151
Investor loans secured by real estate | Special Mention | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	39,360	1,178
Investor loans secured by real estate | Special Mention | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	13,037	0
Investor loans secured by real estate | Special Mention | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		0
Investor loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	4,366	973
Investor loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		2,944
Investor loans secured by real estate | Substandard | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	18,070	1,088
Investor loans secured by real estate | Substandard | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,331	216
Investor loans secured by real estate | Substandard | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		17
Investor loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	8,211	1,623
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,541,577	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	79,595	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,114,050	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	347,932	353,240
Business loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		2,267,278
Business loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	71,668	83,106
Business loans secured by real estate | Pass | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,072,545	1,831,853
Business loans secured by real estate | Pass | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	340,784	352,319
Business loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		13,930
Business loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,337	1,842
Business loans secured by real estate | Special Mention | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	26,263	11,167
Business loans secured by real estate | Special Mention | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	5,180	921
Business loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		6,967
Business loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	6,590	3,433
Business loans secured by real estate | Substandard | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	15,242	3,534
Business loans secured by real estate | Substandard | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,968	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,229,588	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,768,834	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	444,797	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	15,957	17,426
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		1,920,189
Commercial loans | Pass | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,701,772	1,359,662
Commercial loans | Pass | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	402,737	546,594
Commercial loans | Pass | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	12,214	13,933
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		20,641
Commercial loans | Special Mention | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	22,741	13,226
Commercial loans | Special Mention | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	12,335	6,930
Commercial loans | Special Mention | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	1,574	485
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		34,223
Commercial loans | Substandard | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	44,321	20,382
Commercial loans | Substandard | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	29,725	10,833
Commercial loans | Substandard | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	2,169	3,008
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	239,503	305,999
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	232,574	255,024
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	6,929	50,975
Retail loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		305,384
Retail loans | Pass | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	231,917	254,463
Retail loans | Pass | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	6,881	50,921
Retail loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		0
Retail loans | Special Mention | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		0
Retail loans | Special Mention | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		0
Retail loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment		615
Retail loans | Substandard | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	657	561
Retail loans | Substandard | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Gross loans held for investment	$ 48	$ 54