Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Sep. 30, 2023	Dec. 31, 2022
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	$ 13,318,950	$ 14,738,224
Basis adjustment associated with fair value hedge
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	13,318,950	14,738,224
Basis adjustment associated with fair value hedge	48,830	61,926
Current
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	13,308,024	14,694,876
30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,967	20,538
60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	475	185
90+
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	7,484	22,625
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	8,715,596	9,213,516
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,514,056	2,660,321
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	5,719,210	6,112,026
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	444,576	399,034
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	37,754	42,135
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	8,714,419	9,200,307
Investor loans secured by real estate | Current | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,513,631	2,655,892
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	5,719,210	6,103,246
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	444,576	399,034
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	37,002	42,135
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	2,723
Investor loans secured by real estate | 30-59 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	2,723
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	332	0
Investor loans secured by real estate | 60-89 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	332	0
Investor loans secured by real estate | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	845	10,486
Investor loans secured by real estate | 90+ | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	425	4,429
Investor loans secured by real estate | 90+ | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	6,057
Investor loans secured by real estate | 90+ | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Investor loans secured by real estate | 90+ | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	420	0
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,595,921	2,871,588
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	53,668	61,368
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,228,802	2,432,163
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	313,451	378,057
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,590,053	2,855,335
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	52,495	60,177
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,224,107	2,424,174
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	313,451	370,984
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	8,507
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	1,434
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	7,073
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	104
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	104
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Business loans secured by real estate | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	5,868	7,642
Business loans secured by real estate | 90+ | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,173	1,087
Business loans secured by real estate | 90+ | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	4,695	6,555
Business loans secured by real estate | 90+ | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,934,491	2,576,839
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,588,771	2,160,948
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	335,053	404,791
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	10,667	11,100
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,930,611	2,564,012
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,585,718	2,152,302
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	335,053	401,199
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	9,840	10,511
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,966	8,249
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	2,677	4,657
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	3,592
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	289	0
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	143	81
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	143	81
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Commercial loans | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	771	4,497
Commercial loans | 90+ | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	233	3,908
Commercial loans | 90+ | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Commercial loans | 90+ | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	538	589
Retail loans
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	72,942	76,281
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	70,984	72,997
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,958	3,284
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	72,941	75,222
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	70,984	71,940
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1,957	3,282
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1	1,059
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	1,057
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	1	2
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Retail loans | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Retail loans | 90+ | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	0	0
Retail loans | 90+ | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment before basis adjustment	$ 0	$ 0