Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Jun. 30, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Dec. 31, 2019
Loans Held for Investment
Loans held for investment	$ 13,982,861		$ 13,236,433
Allowance for credit losses for loans held for investment	(211,481)	$ (232,774)	(268,018)	$ (282,503)	$ (282,271)	$ (35,698)
Loans held for investment, net	13,771,380		12,968,415
Loans held for sale, at lower of cost or fair value	8,100		601
Net deferred origination fees (costs)	3,000		2,600
Unaccreted mark-to-market discount	85,000		113,800
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	2,504,188		1,947,250
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	8,870,992		8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,823,065		2,675,085
Allowance for credit losses for loans held for investment	(42,467)	(47,112)	(49,176)	(54,105)	(63,007)	(1,899)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	5,705,666		5,171,356
Allowance for credit losses for loans held for investment	(52,164)	(59,059)	(62,534)	(67,336)	(63,511)	(729)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	292,815		321,993
Allowance for credit losses for loans held for investment	(8,017)	(9,548)	(12,435)	(15,557)	(18,804)	(4,484)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	49,446		57,331
Allowance for credit losses for loans held for investment	(3,879)	(4,681)	(5,159)	(5,327)	(2,010)	(1,915)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,666,582		2,541,577
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	69,937		79,595
Allowance for credit losses for loans held for investment	(5,104)	(6,317)	(6,567)	(6,160)	(4,368)	(2,119)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,242,164		2,114,050
Allowance for credit losses for loans held for investment	(33,679)	(35,747)	(50,517)	(48,666)	(48,213)	(2,781)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	354,481		347,932
Allowance for credit losses for loans held for investment	(9,626)	(11,436)	(11,451)	(11,988)	(13,060)	(592)
Commercial loans
Loans Held for Investment
Loans held for investment	2,294,552		2,229,588
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	1,888,870		1,768,834
Allowance for credit losses for loans held for investment	(37,595)	(39,879)	(46,964)	(47,914)	(41,967)	(13,857)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	392,950		444,797
Allowance for credit losses for loans held for investment	(17,518)	(17,313)	(20,525)	(20,149)	(21,676)	(5,816)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	12,732		15,957
Allowance for credit losses for loans held for investment	(632)	(730)	(995)	(951)	(600)	(445)
Retail loans
Loans Held for Investment
Loans held for investment	150,735		239,503
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	144,309		232,574
Allowance for credit losses for loans held for investment	(529)	(670)	(1,204)	(1,243)	(1,479)	(655)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	6,426		6,929
Allowance for credit losses for loans held for investment	$ (271)	$ (282)	$ (491)	$ (3,107)	$ (3,576)	$ (406)