Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Past Due [Line Items]
Total	$ 13,450,840	$ 8,722,311
Current
Financing Receivable, Past Due [Line Items]
Current	13,421,464	8,703,209
30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	7,084	2,104
60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	1,086	10,559
90
Financing Receivable, Past Due [Line Items]
Days Past Due	21,206	6,439
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	8,245,481	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,707,930	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Total	5,142,069	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Total	337,872	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	57,610	68,431
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	8,242,561	4,150,427
Investor loans secured by real estate | Current | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,707,169	2,067,874
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Current	5,142,069	1,575,726
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Current	336,977	438,786
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	56,346	68,041
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	673	1,179
Investor loans secured by real estate | 30-59 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1,179
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	673	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	2,247	1,478
Investor loans secured by real estate | 90 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	761	1,088
Investor loans secured by real estate | 90 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	895	0
Investor loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	591	390
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	2,563,243	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	84,126	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,119,788	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Total	359,329	353,240
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	2,556,929	2,286,409
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	83,116	86,946
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,114,484	1,846,223
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Current	359,329	353,240
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	0	331
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	331
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	250	589
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	589
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	250	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	6,064	846
Business loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	1,010	846
Business loans secured by real estate | 90 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	5,054	0
Business loans secured by real estate | 90 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Total	2,353,723	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Total	1,820,995	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	515,980	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	16,748	17,426
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Current	2,333,955	1,960,382
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Current	1,810,027	1,389,026
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	508,237	555,215
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	15,691	16,141
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	6,037	589
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	5,717	422
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	320	167
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	836	9,968
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	836	826
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	9,142
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	12,895	4,114
Commercial loans | 90 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	4,415	2,996
Commercial loans | 90 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	7,743	0
Commercial loans | 90 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	737	1,118
Retail loans
Financing Receivable, Past Due [Line Items]
Total	288,393	305,999
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Total	243,359	255,024
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Total	45,034	50,975
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Current	288,019	305,991
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Current	242,985	255,024
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Current	45,034	50,967
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	374	5
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	374	0
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	5
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	2
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	2
Retail loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1
Retail loans | 90 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	$ 0	$ 1