Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019
Loans Held for Investment
Loans held for investment	$ 13,594,598		$ 13,236,433
Allowance for credit losses for loans held for investment	(232,774)	$ (266,999)	(268,018)	$ (282,271)	$ (115,422)	$ (35,698)
Loans held for investment, net	13,361,824		12,968,415
Loans held for sale, at lower of cost or fair value	4,714		601
Unaccreted mark-to-market discount	94,400		113,800
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	2,345,364		1,947,250
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	8,700,428		8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,810,233		2,675,085
Allowance for credit losses for loans held for investment	(47,112)	(45,545)	(49,176)	(63,007)	(15,896)	(1,899)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	5,539,464		5,171,356
Allowance for credit losses for loans held for investment	(59,059)	(79,815)	(62,534)	(63,511)	(14,722)	(729)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	297,728		321,993
Allowance for credit losses for loans held for investment	(9,548)	(13,263)	(12,435)	(18,804)	(9,222)	(4,484)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	53,003		57,331
Allowance for credit losses for loans held for investment	(4,681)	(5,141)	(5,159)	(2,010)	(935)	(1,915)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,520,343		2,541,577
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	72,923		79,595
Allowance for credit losses for loans held for investment	(6,317)	(6,451)	(6,567)	(4,368)	(4,044)	(2,119)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,089,300		2,114,050
Allowance for credit losses for loans held for investment	(35,747)	(41,594)	(50,517)	(48,213)	(26,793)	(2,781)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	358,120		347,932
Allowance for credit losses for loans held for investment	(11,436)	(10,876)	(11,451)	(13,060)	(7,503)	(592)
Commercial loans
Loans Held for Investment
Loans held for investment	2,210,359		2,229,588
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	1,795,144		1,768,834
Allowance for credit losses for loans held for investment	(39,879)	(43,373)	(46,964)	(41,967)	(15,742)	(13,857)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	401,315		444,797
Allowance for credit losses for loans held for investment	(17,313)	(18,903)	(20,525)	(21,676)	(16,616)	(5,816)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	13,900		15,957
Allowance for credit losses for loans held for investment	(730)	(890)	(995)	(600)	(516)	(445)
Retail loans
Loans Held for Investment
Loans held for investment	163,468		239,503
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	157,228		232,574
Allowance for credit losses for loans held for investment	(670)	(822)	(1,204)	(1,479)	(1,137)	(655)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	6,240		6,929
Allowance for credit losses for loans held for investment	$ (282)	$ (326)	$ (491)	$ (3,576)	$ (2,296)	$ (406)