Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Dec. 31, 2020
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	$ 1,932,994	$ 1,862,085
Fiscal year before current fiscal year	1,830,360	3,354,750
Two years before current fiscal year	2,963,529	2,140,513
Three years before current fiscal year	1,819,329	1,729,763
Four years before current fiscal year	1,507,457	1,072,603
Prior	2,649,725	2,152,690
Revolving	889,352	912,643
Revolving Converted to Term During the Period	1,852	11,386
Total	13,594,598	13,236,433
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	1,365,536	1,351,350
Fiscal year before current fiscal year	1,350,055	2,386,517
Two years before current fiscal year	2,137,775	1,437,278
Three years before current fiscal year	1,249,591	988,649
Four years before current fiscal year	870,189	674,159
Prior	1,715,122	1,374,935
Revolving	12,160	12,877
Revolving Converted to Term During the Period	0	0
Total	8,700,428	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,810,233	2,675,085
Investor loans secured by real estate | CRE non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	301,013	265,901
Fiscal year before current fiscal year	267,646	541,994
Two years before current fiscal year	477,241	440,351
Three years before current fiscal year	488,498	287,580
Four years before current fiscal year	272,907	279,238
Prior	923,821	791,477
Revolving	9,980	11,114
Revolving Converted to Term During the Period	0	0
Total	2,741,106	2,617,655
Investor loans secured by real estate | CRE non-owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	16,356	6,669
Three years before current fiscal year	3,395	437
Four years before current fiscal year	0	2,516
Prior	17,581	29,738
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	37,332	39,360
Investor loans secured by real estate | CRE non-owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	9,732
Two years before current fiscal year	25,834	2,045
Three years before current fiscal year	0	0
Four years before current fiscal year	0	516
Prior	5,424	5,218
Revolving	537	559
Revolving Converted to Term During the Period	0	0
Total	31,795	18,070
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total	5,539,464	5,171,356
Investor loans secured by real estate | Multifamily | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	1,027,783	1,027,644
Fiscal year before current fiscal year	984,120	1,677,716
Two years before current fiscal year	1,514,852	899,123
Three years before current fiscal year	707,270	665,939
Four years before current fiscal year	570,526	354,859
Prior	727,578	531,287
Revolving	1,643	420
Revolving Converted to Term During the Period	0	0
Total	5,533,772	5,156,988
Investor loans secured by real estate | Multifamily | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	1,758
Two years before current fiscal year	1,745	2,630
Three years before current fiscal year	2,073	0
Four years before current fiscal year	0	8,649
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	3,818	13,037
Investor loans secured by real estate | Multifamily | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	0	0
Three years before current fiscal year	544	559
Four years before current fiscal year	559	772
Prior	771	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	1,874	1,331
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total	297,728	321,993
Investor loans secured by real estate | Construction and land | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	36,740	57,309
Fiscal year before current fiscal year	97,789	144,759
Two years before current fiscal year	93,185	73,313
Three years before current fiscal year	35,334	18,625
Four years before current fiscal year	8,765	20,531
Prior	25,915	6,672
Revolving	0	784
Revolving Converted to Term During the Period	0	0
Total	297,728	321,993
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	53,003	57,331
Investor loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	500	8,306
Two years before current fiscal year	7,391	9,029
Three years before current fiscal year	10,116	13,418
Four years before current fiscal year	12,155	6,305
Prior	11,116	7,696
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	41,278	44,754
Investor loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	496
Fiscal year before current fiscal year	0	1,032
Two years before current fiscal year	0	1,159
Three years before current fiscal year	0	1,000
Four years before current fiscal year	3,038	373
Prior	750	306
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	3,788	4,366
Investor loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	1,220
Two years before current fiscal year	1,171	2,959
Three years before current fiscal year	2,361	1,091
Four years before current fiscal year	2,239	400
Prior	2,166	2,541
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	7,937	8,211
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	338,745	344,058
Fiscal year before current fiscal year	330,353	516,400
Two years before current fiscal year	440,622	422,995
Three years before current fiscal year	345,111	446,733
Four years before current fiscal year	381,622	267,062
Prior	680,837	529,565
Revolving	3,053	14,518
Revolving Converted to Term During the Period	0	246
Total	2,520,343	2,541,577
Business loans secured by real estate | CRE non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year		344,058
Fiscal year before current fiscal year		516,400
Two years before current fiscal year		422,995
Three years before current fiscal year		446,733
Four years before current fiscal year		267,062
Prior		529,565
Revolving		14,518
Revolving Converted to Term During the Period		246
Total		2,541,577
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	72,923	79,595
Business loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	2,502	3,253
Fiscal year before current fiscal year	3,423	7,637
Two years before current fiscal year	7,562	12,608
Three years before current fiscal year	11,398	16,058
Four years before current fiscal year	12,676	8,488
Prior	27,290	23,624
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	64,851	71,668
Business loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	0	1,200
Three years before current fiscal year	0	0
Four years before current fiscal year	0	137
Prior	150	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	150	1,337
Business loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	0	184
Three years before current fiscal year	1,336	1,987
Four years before current fiscal year	2,160	1,376
Prior	4,426	3,043
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	7,922	6,590
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,089,300	2,114,050
Business loans secured by real estate | CRE owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	276,707	293,324
Fiscal year before current fiscal year	281,668	409,758
Two years before current fiscal year	355,162	332,672
Three years before current fiscal year	269,806	327,475
Four years before current fiscal year	278,179	225,098
Prior	596,013	469,704
Revolving	3,053	14,268
Revolving Converted to Term During the Period	0	246
Total	2,060,588	2,072,545
Business loans secured by real estate | CRE owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	2,190
Fiscal year before current fiscal year	0	15,917
Two years before current fiscal year	6,086	3,802
Three years before current fiscal year	1,741	0
Four years before current fiscal year	3,043	4,153
Prior	0	201
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	10,870	26,263
Business loans secured by real estate | CRE owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	0	3,636
Three years before current fiscal year	6,151	4,214
Four years before current fiscal year	5,873	1,169
Prior	5,818	5,973
Revolving	0	250
Revolving Converted to Term During the Period	0	0
Total	17,842	15,242
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	358,120	347,932
Business loans secured by real estate | Franchise real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	59,536	44,413
Fiscal year before current fiscal year	44,384	81,438
Two years before current fiscal year	71,812	66,241
Three years before current fiscal year	54,679	96,999
Four years before current fiscal year	79,691	24,673
Prior	47,140	27,020
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	357,242	340,784
Business loans secured by real estate | Franchise real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	878
Fiscal year before current fiscal year	878	1,650
Two years before current fiscal year	0	2,652
Three years before current fiscal year	0	0
Four years before current fiscal year	0	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	878	5,180
Business loans secured by real estate | Franchise real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year		0
Fiscal year before current fiscal year		0
Two years before current fiscal year		0
Three years before current fiscal year		0
Four years before current fiscal year		1,968
Prior		0
Revolving		0
Revolving Converted to Term During the Period		0
Total		1,968
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	215,357	155,831
Fiscal year before current fiscal year	143,142	444,000
Two years before current fiscal year	382,538	266,221
Three years before current fiscal year	222,333	280,317
Four years before current fiscal year	246,838	97,412
Prior	149,643	120,099
Revolving	848,656	854,568
Revolving Converted to Term During the Period	1,852	11,140
Total	2,210,359	2,229,588
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,795,144	1,768,834
Commercial loans | Commercial and industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	158,312	127,082
Fiscal year before current fiscal year	114,467	260,368
Two years before current fiscal year	243,248	159,001
Three years before current fiscal year	136,513	210,163
Four years before current fiscal year	189,735	51,800
Prior	95,416	82,291
Revolving	805,860	801,752
Revolving Converted to Term During the Period	1,852	9,315
Total	1,745,403	1,701,772
Commercial loans | Commercial and industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	735
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	231	2,331
Three years before current fiscal year	0	185
Four years before current fiscal year	0	1,320
Prior	0	243
Revolving	13,558	17,890
Revolving Converted to Term During the Period	0	37
Total	13,789	22,741
Commercial loans | Commercial and industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	1,940	3,310
Two years before current fiscal year	1,231	2,737
Three years before current fiscal year	3,334	610
Four years before current fiscal year	33	1,333
Prior	2,365	2,446
Revolving	27,049	32,858
Revolving Converted to Term During the Period	0	1,027
Total	35,952	44,321
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	401,315	444,797
Commercial loans | Franchise non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	56,892	27,607
Fiscal year before current fiscal year	26,322	164,025
Two years before current fiscal year	133,288	94,494
Three years before current fiscal year	76,113	46,174
Four years before current fiscal year	35,506	40,829
Prior	45,833	27,745
Revolving	1,512	1,361
Revolving Converted to Term During the Period	0	502
Total	375,466	402,737
Commercial loans | Franchise non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year		0
Fiscal year before current fiscal year		7,267
Two years before current fiscal year		2,037
Three years before current fiscal year		230
Four years before current fiscal year		480
Prior		2,321
Revolving		0
Revolving Converted to Term During the Period		0
Total		12,335
Commercial loans | Franchise non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	6,690
Two years before current fiscal year	2,293	3,706
Three years before current fiscal year	4,559	18,425
Four years before current fiscal year	17,749	700
Prior	1,248	204
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	25,849	29,725
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	13,900	15,957
Commercial loans | SBA non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	153	407
Fiscal year before current fiscal year	413	2,257
Two years before current fiscal year	2,167	1,558
Three years before current fiscal year	1,469	2,674
Four years before current fiscal year	3,557	610
Prior	4,060	4,449
Revolving	0	0
Revolving Converted to Term During the Period	0	259
Total	11,819	12,214
Commercial loans | SBA non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year		0
Fiscal year before current fiscal year		0
Two years before current fiscal year		0
Three years before current fiscal year		1,574
Four years before current fiscal year		0
Prior		0
Revolving		0
Revolving Converted to Term During the Period		0
Total		1,574
Commercial loans | SBA non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	83
Two years before current fiscal year	80	357
Three years before current fiscal year	345	282
Four years before current fiscal year	258	340
Prior	721	400
Revolving	677	707
Revolving Converted to Term During the Period	0	0
Total	2,081	2,169
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	13,356	10,846
Fiscal year before current fiscal year	6,810	7,833
Two years before current fiscal year	2,594	14,019
Three years before current fiscal year	2,294	14,064
Four years before current fiscal year	8,808	33,970
Prior	104,123	128,091
Revolving	25,483	30,680
Revolving Converted to Term During the Period	0	0
Total	163,468	239,503
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total	157,228	232,574
Retail loans | Single family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	13,312	10,794
Fiscal year before current fiscal year	6,771	7,714
Two years before current fiscal year	2,524	13,982
Three years before current fiscal year	2,271	14,039
Four years before current fiscal year	8,789	33,968
Prior	101,020	124,248
Revolving	22,487	27,172
Revolving Converted to Term During the Period	0	0
Total	157,174	231,917
Retail loans | Single family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	0
Two years before current fiscal year	0	0
Three years before current fiscal year	0	0
Four years before current fiscal year	0	0
Prior	54	657
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	54	657
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total	6,240	6,929
Retail loans | Consumer | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	44	52
Fiscal year before current fiscal year	39	112
Two years before current fiscal year	63	37
Three years before current fiscal year	23	25
Four years before current fiscal year	19	2
Prior	3,012	3,145
Revolving	2,996	3,508
Revolving Converted to Term During the Period	0	0
Total	6,196	6,881
Retail loans | Consumer | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Current fiscal year	0	0
Fiscal year before current fiscal year	0	7
Two years before current fiscal year	7	0
Three years before current fiscal year	0	0
Four years before current fiscal year	0	0
Prior	37	41
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	$ 44	$ 48