Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Loans Held for Investment
Loans held for investment	$ 14,676,298	$ 14,295,897
Loans held for investment before basis adjustment	14,738,224
Allowance for credit losses for loans held for investment	(195,651)	(197,752)	$ (268,018)	$ (35,698)
Loans held for investment, net	14,480,647	14,098,145
Loans held for sale, at lower of cost or fair value	2,643	10,869
Net deferred origination fees (costs)	1,900	3,500
Unaccreted mark-to-market discount	54,800	77,100
Basis adjustment associated with fair value hedge
Loans Held for Investment
Loans held for investment before basis adjustment	14,738,224	14,295,897
Basis adjustment associated with fair value hedge	(61,926)	0
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	2,489,203	2,507,911
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	9,213,516	8,987,628
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,660,321	2,771,137
Allowance for credit losses for loans held for investment	(33,692)	(37,380)	(49,176)	(1,899)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	6,112,026	5,891,934
Allowance for credit losses for loans held for investment	(56,334)	(55,209)	(62,534)	(729)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	399,034	277,640
Allowance for credit losses for loans held for investment	(7,114)	(5,211)	(12,435)	(4,484)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	42,135	46,917
Allowance for credit losses for loans held for investment	(2,592)	(3,201)	(5,159)	(1,915)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,871,588	2,700,579
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	61,368	69,184
Allowance for credit losses for loans held for investment	(4,348)	(4,866)	(6,567)	(2,119)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,432,163	2,251,014
Allowance for credit losses for loans held for investment	(32,340)	(29,575)	(50,517)	(2,781)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	378,057	380,381
Allowance for credit losses for loans held for investment	(7,019)	(7,985)	(11,451)	(592)
Commercial loans
Loans Held for Investment
Loans held for investment	2,576,839	2,506,733
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	2,160,948	2,103,112
Allowance for credit losses for loans held for investment	(35,169)	(38,136)	(46,964)	(13,857)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	404,791	392,576
Allowance for credit losses for loans held for investment	(16,029)	(15,084)	(20,525)	(5,816)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	11,100	11,045
Allowance for credit losses for loans held for investment	(441)	(565)	(995)	(445)
Retail loans
Loans Held for Investment
Loans held for investment	76,281	100,957
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	72,997	95,292
Allowance for credit losses for loans held for investment	(352)	(255)	(1,204)	(655)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	3,284	5,665
Allowance for credit losses for loans held for investment	$ (221)	$ (285)	$ (491)	$ (406)