Loans - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Jan. 01, 2020	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Loans Held for Investment
Gross loans held for investment	$ 13,236,433		$ 8,722,311
Allowance for credit losses for loans held for investment	(268,018)		(35,698)	$ (36,072)	$ (28,936)
Loans held for investment, net	12,968,415		8,686,613
Loans held for sale, at lower of cost or fair value	601		1,672
Unaccreted mark-to-market discount	113,800		40,700
Investor loans secured by real estate
Loans Held for Investment
Gross loans held for investment	8,225,765		4,153,084
Allowance for credit losses for loans held for investment		$ (9,027)
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Gross loans held for investment	2,675,085		2,070,141
Allowance for credit losses for loans held for investment	(49,176)		(1,899)	(1,624)	(1,273)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Gross loans held for investment	5,171,356		1,575,726
Allowance for credit losses for loans held for investment	(62,534)		(729)	(740)	(614)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Gross loans held for investment	321,993		438,786
Allowance for credit losses for loans held for investment	(12,435)		(4,484)	(5,964)	(5,565)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Gross loans held for investment	57,331		68,431
Allowance for credit losses for loans held for investment	(5,159)		(1,915)	(1,827)	(1,396)
Business loans secured by real estate
Loans Held for Investment
Gross loans held for investment	2,541,577		2,288,175
Allowance for credit losses for loans held for investment		(5,492)
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Gross loans held for investment	79,595		88,381
Allowance for credit losses for loans held for investment	(6,567)		(2,119)	(1,824)	(901)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Gross loans held for investment	2,114,050		1,846,554
Allowance for credit losses for loans held for investment	(50,517)		(2,781)	(1,908)	(923)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Gross loans held for investment	347,932		353,240
Allowance for credit losses for loans held for investment	(11,451)		(592)	(743)	(602)
Commercial loans
Loans Held for Investment
Gross loans held for investment	2,229,588		1,975,053
Allowance for credit losses for loans held for investment		(20,118)
Commercial loans | Commercial and industrial
Loans Held for Investment
Gross loans held for investment	1,768,834		1,393,270
Allowance for credit losses for loans held for investment	(46,964)		(13,857)	(13,695)	(11,018)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Gross loans held for investment	444,797		564,357
Allowance for credit losses for loans held for investment	(20,525)		(5,816)	(6,066)	(5,191)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Gross loans held for investment	15,957		17,426
Allowance for credit losses for loans held for investment	(995)		(445)	(654)	(594)
Retail loans
Loans Held for Investment
Gross loans held for investment	239,503		305,999
Allowance for credit losses for loans held for investment		$ (1,061)
Retail loans | Single family residential
Loans Held for Investment
Gross loans held for investment	232,574		255,024
Allowance for credit losses for loans held for investment	(1,204)		(655)	(808)	(804)
Retail loans | Consumer
Loans Held for Investment
Gross loans held for investment	6,929		50,975
Allowance for credit losses for loans held for investment	$ (491)		$ (406)	$ (219)	$ (55)