Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
2020	$ 1,212,344
2019	3,540,465
2018	2,369,668
2017	1,982,912
2016	1,171,944
Prior	2,385,545
Revolving	779,125
Revolving Converted to Term During the Period	8,837
Total	13,450,840	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	854,784
2019	2,458,374
2018	1,560,916
2017	1,114,295
2016	730,632
Prior	1,513,942
Revolving	12,538
Revolving Converted to Term During the Period	0
Total	8,245,481	4,153,084
Investor loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		4,147,989
Investor loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,151
Investor loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,944
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,707,930	2,070,141
Investor loans secured by real estate | CRE non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	190,701
2019	566,855
2018	475,363
2017	314,190
2016	290,179
Prior	852,044
Revolving	11,031
Revolving Converted to Term During the Period	0
Total	2,700,363	2,067,875
Investor loans secured by real estate | CRE non-owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	1,839
2017	435
2016	0
Prior	1,816
Revolving	0
Revolving Converted to Term During the Period	0
Total	4,090	1,178
Investor loans secured by real estate | CRE non-owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	202
2017	0
2016	517
Prior	2,199
Revolving	559
Revolving Converted to Term During the Period	0
Total	3,477	1,088
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total	5,142,069	1,575,726
Investor loans secured by real estate | Multifamily | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	638,349
2019	1,735,235
2018	964,740
2017	750,548
2016	412,893
Prior	639,171
Revolving	574
Revolving Converted to Term During the Period	0
Total	5,141,510	1,575,510
Investor loans secured by real estate | Multifamily | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Investor loans secured by real estate | Multifamily | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	559
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	559	216
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total	337,872	438,786
Investor loans secured by real estate | Construction and land | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	25,240
2019	145,709
2018	104,702
2017	33,023
2016	19,932
Prior	7,997
Revolving	374
Revolving Converted to Term During the Period	0
Total	336,977	438,769
Investor loans secured by real estate | Construction and land | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Investor loans secured by real estate | Construction and land | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	895
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	895	17
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	57,610	68,431
Investor loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	494
2019	10,412
2018	11,058
2017	14,842
2016	6,712
Prior	9,037
Revolving	0
Revolving Converted to Term During the Period	0
Total	52,555	65,835
Investor loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		973
Investor loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	163
2018	2,117
2017	698
2016	399
Prior	1,678
Revolving	0
Revolving Converted to Term During the Period	0
Total	5,055	1,623
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	233,812
2019	523,420
2018	450,328
2017	479,759
2016	283,961
Prior	587,025
Revolving	4,938
Revolving Converted to Term During the Period	0
Total	2,563,243	2,288,175
Business loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,267,278
Business loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		13,930
Business loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		6,967
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	84,126	88,381
Business loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	2,643
2019	7,658
2018	14,054
2017	17,110
2016	9,643
Prior	26,768
Revolving	95
Revolving Converted to Term During the Period	0
Total	77,971	83,106
Business loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		1,842
Business loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	22
2017	1,994
2016	914
Prior	3,225
Revolving	0
Revolving Converted to Term During the Period	0
Total	6,155	3,433
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,119,788	1,846,554
Business loans secured by real estate | CRE owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	210,662
2019	412,186
2018	358,157
2017	346,906
2016	235,040
Prior	505,999
Revolving	4,593
Revolving Converted to Term During the Period	0
Total	2,073,543	1,831,853
Business loans secured by real estate | CRE owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	15,827
2018	0
2017	10,268
2016	4,186
Prior	2,008
Revolving	0
Revolving Converted to Term During the Period	0
Total	32,289	11,167
Business loans secured by real estate | CRE owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	3,636
2017	725
2016	2,666
Prior	6,679
Revolving	250
Revolving Converted to Term During the Period	0
Total	13,956	3,534
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	359,329	353,240
Business loans secured by real estate | Franchise real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	20,507
2019	87,749
2018	74,459
2017	102,756
2016	31,512
Prior	42,346
Revolving	0
Revolving Converted to Term During the Period	0
Total	359,329	352,319
Business loans secured by real estate | Franchise real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		921
Business loans secured by real estate | Franchise real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	119,216
2019	550,144
2018	343,392
2017	335,937
2016	122,265
Prior	147,157
Revolving	726,775
Revolving Converted to Term During the Period	8,837
Total	2,353,723	1,975,053
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		1,920,189
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		20,641
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		34,223
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,820,995	1,393,270
Commercial loans | Commercial and industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	98,656
2019	339,144
2018	183,259
2017	232,648
2016	71,414
Prior	94,220
Revolving	682,523
Revolving Converted to Term During the Period	6,311
Total	1,708,175	1,359,662
Commercial loans | Commercial and industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	43
2018	17,231
2017	15,810
2016	1,398
Prior	0
Revolving	7,886
Revolving Converted to Term During the Period	0
Total	42,368	13,226
Commercial loans | Commercial and industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	4,902
2018	22,763
2017	1,111
2016	1,265
Prior	4,393
Revolving	34,271
Revolving Converted to Term During the Period	1,747
Total	70,452	20,382
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	515,980	564,357
Commercial loans | Franchise non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	20,205
2019	201,621
2018	117,113
2017	53,160
2016	47,290
Prior	43,239
Revolving	1,361
Revolving Converted to Term During the Period	511
Total	484,500	546,594
Commercial loans | Franchise non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		6,930
Commercial loans | Franchise non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	2,050
2018	957
2017	28,473
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	31,480	10,833
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	16,748	17,426
Commercial loans | SBA non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	355
2019	2,299
2018	1,700
2017	2,254
2016	623
Prior	3,881
Revolving	0
Revolving Converted to Term During the Period	268
Total	11,380	13,933
Commercial loans | SBA non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,661
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,661	485
Commercial loans | SBA non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	85
2018	369
2017	820
2016	275
Prior	1,424
Revolving	734
Revolving Converted to Term During the Period	0
Total	3,707	3,008
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	4,532
2019	8,527
2018	15,032
2017	52,921
2016	35,086
Prior	137,421
Revolving	34,874
Revolving Converted to Term During the Period	0
Total	288,393	305,999
Retail loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		305,384
Retail loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Retail loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		615
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total	243,359	255,024
Retail loans | Single family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	4,469
2019	8,385
2018	14,972
2017	14,884
2016	35,074
Prior	133,137
Revolving	31,452
Revolving Converted to Term During the Period	0
Total	242,373	254,463
Retail loans | Single family residential | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	57
Revolving	0
Revolving Converted to Term During the Period	0
Total	57	0
Retail loans | Single family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	929
Revolving	0
Revolving Converted to Term During the Period	0
Total	929	561
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total	45,034	50,975
Retail loans | Consumer | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	63
2019	142
2018	60
2017	38,037
2016	12
Prior	3,255
Revolving	3,422
Revolving Converted to Term During the Period	0
Total	44,991	50,921
Retail loans | Consumer | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Retail loans | Consumer | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	43
Revolving	0
Revolving Converted to Term During the Period	0
Total	$ 43	$ 54