Loans Held for Investment - Internal Risk Grading System (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	$ 13,450,840	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	8,245,481	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,707,930	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	5,142,069	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	337,872	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	57,610	68,431
Investor loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		4,147,989
Investor loans secured by real estate | Pass | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,700,363	2,067,875
Investor loans secured by real estate | Pass | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	5,141,510	1,575,510
Investor loans secured by real estate | Pass | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	336,977	438,769
Investor loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	52,555	65,835
Investor loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,151
Investor loans secured by real estate | Special Mention | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	4,090	1,178
Investor loans secured by real estate | Special Mention | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Investor loans secured by real estate | Special Mention | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Investor loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		973
Investor loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,944
Investor loans secured by real estate | Substandard | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	3,477	1,088
Investor loans secured by real estate | Substandard | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	559	216
Investor loans secured by real estate | Substandard | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	895	17
Investor loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	5,055	1,623
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,563,243	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	84,126	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,119,788	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	359,329	353,240
Business loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,267,278
Business loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	77,971	83,106
Business loans secured by real estate | Pass | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,073,543	1,831,853
Business loans secured by real estate | Pass | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	359,329	352,319
Business loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		13,930
Business loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		1,842
Business loans secured by real estate | Special Mention | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	32,289	11,167
Business loans secured by real estate | Special Mention | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		921
Business loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		6,967
Business loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	6,155	3,433
Business loans secured by real estate | Substandard | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	13,956	3,534
Business loans secured by real estate | Substandard | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,353,723	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,820,995	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	515,980	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	16,748	17,426
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		1,920,189
Commercial loans | Pass | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,708,175	1,359,662
Commercial loans | Pass | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	484,500	546,594
Commercial loans | Pass | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	11,380	13,933
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		20,641
Commercial loans | Special Mention | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	42,368	13,226
Commercial loans | Special Mention | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		6,930
Commercial loans | Special Mention | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,661	485
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		34,223
Commercial loans | Substandard | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	70,452	20,382
Commercial loans | Substandard | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	31,480	10,833
Commercial loans | Substandard | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	3,707	3,008
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	288,393	305,999
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	243,359	255,024
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	45,034	50,975
Retail loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		305,384
Retail loans | Pass | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	242,373	254,463
Retail loans | Pass | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	44,991	50,921
Retail loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Retail loans | Special Mention | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	57	0
Retail loans | Special Mention | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Retail loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		615
Retail loans | Substandard | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	929	561
Retail loans | Substandard | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	$ 43	$ 54