Loans Held for Investment - Combination of Loans Modification Without Troubled Debt Restructuring Classification (Details) - COVID-19 Related Loan Modifications $ in Thousands	Jun. 30, 2020 USD ($)
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	$ 2,244,974
Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	1,645,274
Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	599,700
Investor loans secured by real estate
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	1,288,774
Investor loans secured by real estate | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	970,468
Investor loans secured by real estate | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	318,306
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	796,514
Investor loans secured by real estate | CRE non-owner-occupied | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	545,928
Investor loans secured by real estate | CRE non-owner-occupied | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	250,586
Investor loans secured by real estate | Multifamily
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	492,260
Investor loans secured by real estate | Multifamily | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	424,540
Investor loans secured by real estate | Multifamily | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	67,720
Business loans secured by real estate
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	537,864
Business loans secured by real estate | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	413,362
Business loans secured by real estate | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	124,502
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	363,104
Business loans secured by real estate | CRE owner-occupied | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	297,662
Business loans secured by real estate | CRE owner-occupied | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	65,442
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	174,734
Business loans secured by real estate | Franchise real estate secured | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	115,674
Business loans secured by real estate | Franchise real estate secured | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	59,060
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	26
Business loans secured by real estate | SBA secured by real estate | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	26
Business loans secured by real estate | SBA secured by real estate | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	0
Commercial loans
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	406,672
Commercial loans | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	250,045
Commercial loans | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	156,627
Commercial loans | Commercial and industrial
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	91,558
Commercial loans | Commercial and industrial | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	61,497
Commercial loans | Commercial and industrial | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	30,061
Commercial loans | Franchise non-real estate secured
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	315,114
Commercial loans | Franchise non-real estate secured | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	188,548
Commercial loans | Franchise non-real estate secured | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	126,566
Retail loans
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	11,664
Retail loans | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	11,399
Retail loans | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	265
Retail loans | Single family residential
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	11,656
Retail loans | Single family residential | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	11,391
Retail loans | Single family residential | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	265
Retail loans | Consumer
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	8
Retail loans | Consumer | Full payment Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	8
Retail loans | Consumer | Interest-only Deferral
Financing Receivable, Troubled Debt Restructuring [Line Items]
Total balance of loans modified	$ 0