Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Financing Receivable, Past Due [Line Items]
Loans held for investment	$ 14,295,897	$ 13,236,433
Current
Financing Receivable, Past Due [Line Items]
Loans held for investment	14,276,402	13,223,111
30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment	1,395	1,269
60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	57
90+
Financing Receivable, Past Due [Line Items]
Loans held for investment	18,100	11,996
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	8,987,628	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,771,137	2,675,085
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment	5,891,934	5,171,356
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment	277,640	321,993
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	46,917	57,331
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment	8,975,806	8,223,750
Investor loans secured by real estate | Current | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,760,882	2,674,328
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment	5,890,704	5,171,355
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment	277,640	321,993
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	46,580	56,074
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment	1,230	1
Investor loans secured by real estate | 30-59 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment	1,230	1
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 60-89 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment	10,592	2,014
Investor loans secured by real estate | 90+ | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	10,255	757
Investor loans secured by real estate | 90+ | Multifamily
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 90+ | Construction and land
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Investor loans secured by real estate | 90+ | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	337	1,257
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,700,579	2,541,577
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	69,184	79,595
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,251,014	2,114,050
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	380,381	347,932
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,695,186	2,534,714
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	68,743	78,036
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,246,062	2,108,746
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	380,381	347,932
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	486
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	486
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Business loans secured by real estate | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment	5,393	6,377
Business loans secured by real estate | 90+ | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Loans held for investment	441	1,073
Business loans secured by real estate | 90+ | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Loans held for investment	4,952	5,304
Business loans secured by real estate | 90+ | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,506,733	2,229,588
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,103,112	1,768,834
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	392,576	444,797
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	11,045	15,957
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,504,453	2,225,160
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,101,558	1,765,451
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	392,576	444,797
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	10,319	14,912
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment	165	766
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment	92	428
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	73	338
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	57
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	57
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Commercial loans | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment	2,115	3,605
Commercial loans | 90+ | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Loans held for investment	1,462	2,898
Commercial loans | 90+ | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Commercial loans | 90+ | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Loans held for investment	653	707
Retail loans
Financing Receivable, Past Due [Line Items]
Loans held for investment	100,957	239,503
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment	95,292	232,574
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment	5,665	6,929
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Loans held for investment	100,957	239,487
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment	95,292	232,559
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment	5,665	6,928
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	16
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	15
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	1
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Retail loans | 90+
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Retail loans | 90+ | Single family residential
Financing Receivable, Past Due [Line Items]
Loans held for investment	0	0
Retail loans | 90+ | Consumer
Financing Receivable, Past Due [Line Items]
Loans held for investment	$ 0	$ 0