Loans Held for Investment - Schedule of Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Loans Held for Investment
Loans held for investment	$ 13,289,020	$ 14,676,298
Loans held for investment before basis adjustment	13,318,571	14,738,224
Allowance for credit losses for loans held for investment	(192,471)	(195,651)	$ (197,752)	$ (268,018)
Loans held for investment, net	13,096,549	14,480,647
Loans held for sale, at lower of cost or fair value	0	2,643
Net deferred origination fees	(74)	(1,900)
Unaccreted mark-to-market discount	43,300	54,800
Basis adjustment associated with fair value hedge
Loans Held for Investment
Loans held for investment before basis adjustment	13,318,571	14,738,224
Basis adjustment associated with fair value hedge	(29,551)	(61,926)
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	1,703,470	2,489,203
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	8,576,026	9,213,516
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,421,772	2,660,321
Allowance for credit losses for loans held for investment	(31,030)	(33,692)	(37,380)	(49,176)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	5,645,310	6,112,026
Allowance for credit losses for loans held for investment	(56,312)	(56,334)	(55,209)	(62,534)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	472,544	399,034
Allowance for credit losses for loans held for investment	(9,314)	(7,114)	(5,211)	(12,435)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	36,400	42,135
Allowance for credit losses for loans held for investment	(2,182)	(2,592)	(3,201)	(5,159)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,546,589	2,871,588
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	50,741	61,368
Allowance for credit losses for loans held for investment	(4,427)	(4,348)	(4,866)	(6,567)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,191,334	2,432,163
Allowance for credit losses for loans held for investment	(28,787)	(32,340)	(29,575)	(50,517)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	304,514	378,057
Allowance for credit losses for loans held for investment	(7,499)	(7,019)	(7,985)	(11,451)
Commercial loans
Loans Held for Investment
Loans held for investment	2,121,255	2,576,839
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	1,790,608	2,160,948
Allowance for credit losses for loans held for investment	(36,692)	(35,169)	(38,136)	(46,964)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	319,721	404,791
Allowance for credit losses for loans held for investment	(15,131)	(16,029)	(15,084)	(20,525)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	10,926	11,100
Allowance for credit losses for loans held for investment	(458)	(441)	(565)	(995)
Retail loans
Loans Held for Investment
Loans held for investment	74,701	76,281
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	72,752	72,997
Allowance for credit losses for loans held for investment	(505)	(352)	(255)	(1,204)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	1,949	3,284
Allowance for credit losses for loans held for investment	$ (134)	$ (221)	$ (285)	$ (491)