Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Jun. 30, 2020	Jan. 01, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Dec. 31, 2018
Loans Held for Investment
Loans held for investment	$ 13,450,840			$ 8,722,311
Allowance for credit losses for loans held for investment	(282,503)	$ (282,271)		(35,698)	$ (35,000)	$ (35,026)	$ (36,072)
Loans held for investment, net	13,168,337			8,686,613
Loans held for sale, at lower of cost or fair value	1,032			1,672
Unaccreted mark-to-market discount	126,300			40,700
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	8,245,481			4,153,084
Allowance for credit losses for loans held for investment			$ (9,027)
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,707,930			2,070,141
Allowance for credit losses for loans held for investment	(54,105)	(63,007)		(1,899)	(1,878)	(1,765)	(1,624)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	5,142,069			1,575,726
Allowance for credit losses for loans held for investment	(67,336)	(63,511)		(729)	(715)	(705)	(740)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	337,872			438,786
Allowance for credit losses for loans held for investment	(15,557)	(18,804)		(4,484)	(4,791)	(5,408)	(5,964)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	57,610			68,431
Allowance for credit losses for loans held for investment	(5,327)	(2,010)		(1,915)	(1,790)	(1,322)	(1,827)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,563,243			2,288,175
Allowance for credit losses for loans held for investment			(5,492)
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	84,126			88,381
Allowance for credit losses for loans held for investment	(6,160)	(4,368)		(2,119)	(2,110)	(1,611)	(1,824)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,119,788			1,846,554
Allowance for credit losses for loans held for investment	(48,666)	(48,213)		(2,781)	(2,564)	(2,299)	(1,908)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	359,329			353,240
Allowance for credit losses for loans held for investment	(11,988)	(13,060)		(592)	(567)	(579)	(743)
Commercial loans
Loans Held for Investment
Loans held for investment	2,353,723			1,975,053
Allowance for credit losses for loans held for investment			(20,118)
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	1,820,995			1,393,270
Allowance for credit losses for loans held for investment	(47,914)	(41,967)		(13,857)	(12,896)	(13,796)	(13,695)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	515,980			564,357
Allowance for credit losses for loans held for investment	(20,149)	(21,676)		(5,816)	(6,166)	(6,186)	(6,066)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	16,748			17,426
Allowance for credit losses for loans held for investment	(951)	(600)		(445)	(481)	(430)	(654)
Retail loans
Loans Held for Investment
Loans held for investment	288,393			305,999
Allowance for credit losses for loans held for investment			$ (1,061)
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	243,359			255,024
Allowance for credit losses for loans held for investment	(1,243)	(1,479)		(655)	(691)	(704)	(808)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	45,034			50,975
Allowance for credit losses for loans held for investment	$ (3,107)	$ (3,576)		$ (406)	$ (351)	$ (221)	$ (219)