Allowance for Credit Losses - Amortized Cost of Loans by Estimated LTV and FICO Bands (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	$ 2,183,956
2019	3,628,848
2018	2,540,813
2017	1,998,126
2016	1,281,491
Prior	2,541,780
Revolving	901,170
Revolving Converted to Term During the Period	6,700
Total	15,082,884	$ 8,722,311
Investor loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	692,143
2019	2,473,310
2018	1,661,597
2017	1,175,829
2016	802,350
Prior	1,607,953
Revolving	12,975
Revolving Converted to Term During the Period	0
Total	8,426,157	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,783,692	2,070,141
Investor loans secured by real estate | CRE non-owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	73,769
2019	241,867
2018	191,322
2017	156,622
2016	189,946
Prior	618,498
Revolving	10,924
Revolving Converted to Term During the Period	0
Total	1,482,948
Investor loans secured by real estate | CRE non-owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	54,874
2019	214,909
2018	119,539
2017	177,366
2016	104,448
Prior	231,545
Revolving	559
Revolving Converted to Term During the Period	0
Total	903,240
Investor loans secured by real estate | CRE non-owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	22,526
2019	108,464
2018	169,089
2017	23,368
2016	25,379
Prior	31,566
Revolving	0
Revolving Converted to Term During the Period	0
Total	380,392
Investor loans secured by real estate | CRE non-owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	4,260
2019	2,749
2018	2,939
2017	2,345
2016	213
Prior	4,606
Revolving	0
Revolving Converted to Term During the Period	0
Total	17,112
Investor loans secured by real estate | Multifamily
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	5,225,557	1,575,726
Investor loans secured by real estate | Multifamily | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	123,726
2019	387,468
2018	341,633
2017	223,446
2016	95,645
Prior	290,333
Revolving	599
Revolving Converted to Term During the Period	0
Total	1,462,850
Investor loans secured by real estate | Multifamily | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	164,019
2019	734,321
2018	386,341
2017	266,322
2016	167,271
Prior	283,527
Revolving	432
Revolving Converted to Term During the Period	0
Total	2,002,233
Investor loans secured by real estate | Multifamily | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	234,636
2019	609,946
2018	270,888
2017	269,910
2016	211,919
Prior	122,199
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,719,498
Investor loans secured by real estate | Multifamily | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	22,034
2018	11,185
2017	1,898
2016	0
Prior	5,859
Revolving	0
Revolving Converted to Term During the Period	0
Total	40,976
Investor loans secured by real estate | Construction and land
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	357,426	438,786
Investor loans secured by real estate | Construction and land | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	13,838
2019	136,216
2018	107,812
2017	30,148
2016	0
Prior	8,468
Revolving	461
Revolving Converted to Term During the Period	0
Total	296,943
Investor loans secured by real estate | Construction and land | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	4,632
2018	42,778
2017	8,223
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	55,633
Investor loans secured by real estate | Construction and land | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	3,697
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	3,697
Investor loans secured by real estate | Construction and land | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	1,153
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,153
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	59,482	68,431
Investor loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	1,148
2018	655
2017	841
2016	332
Prior	436
Revolving	0
Revolving Converted to Term During the Period	0
Total	3,412
Investor loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	3,187
2018	1,647
2017	3,860
2016	623
Prior	4,791
Revolving	0
Revolving Converted to Term During the Period	0
Total	14,108
Investor loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	495
2019	3,724
2018	7,802
2017	5,364
2016	4,358
Prior	2,829
Revolving	0
Revolving Converted to Term During the Period	0
Total	24,572
Investor loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	2,645
2018	3,117
2017	6,116
2016	2,216
Prior	3,296
Revolving	0
Revolving Converted to Term During the Period	0
Total	17,390
Business loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	240,336
2019	548,159
2018	443,266
2017	453,579
2016	310,433
Prior	619,080
Revolving	5,490
Revolving Converted to Term During the Period	0
Total	2,620,343	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	85,542	88,381
Business loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	736
2019	1,735
2018	6,537
2017	4,784
2016	2,348
Prior	10,568
Revolving	365
Revolving Converted to Term During the Period	0
Total	27,073
Business loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	104
2019	514
2018	2,337
2017	2,643
2016	2,252
Prior	4,121
Revolving	0
Revolving Converted to Term During the Period	0
Total	11,971
Business loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	264
2019	2,687
2018	754
2017	4,602
2016	3,167
Prior	5,871
Revolving	0
Revolving Converted to Term During the Period	0
Total	17,345
Business loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	801
2019	2,757
2018	4,531
2017	7,195
2016	3,234
Prior	10,635
Revolving	0
Revolving Converted to Term During the Period	0
Total	29,153
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,170,154	1,846,554
Business loans secured by real estate | CRE owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	55,579
2019	153,995
2018	171,876
2017	196,583
2016	150,233
Prior	377,115
Revolving	5,125
Revolving Converted to Term During the Period	0
Total	1,110,506
Business loans secured by real estate | CRE owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	63,067
2019	100,226
2018	89,960
2017	64,803
2016	73,724
Prior	81,403
Revolving	0
Revolving Converted to Term During the Period	0
Total	473,183
Business loans secured by real estate | CRE owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	56,327
2019	175,639
2018	63,734
2017	54,957
2016	39,714
Prior	58,581
Revolving	0
Revolving Converted to Term During the Period	0
Total	448,952
Business loans secured by real estate | CRE owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	43,052
2019	21,798
2018	27,345
2017	14,164
2016	4,183
Prior	26,971
Revolving	0
Revolving Converted to Term During the Period	0
Total	137,513
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	364,647	353,240
Business loans secured by real estate | Franchise real estate secured | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	7,461
2019	18,360
2018	14,615
2017	16,057
2016	11,567
Prior	20,926
Revolving	0
Revolving Converted to Term During the Period	0
Total	88,986
Business loans secured by real estate | Franchise real estate secured | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	928
2019	8,900
2018	13,096
2017	29,110
2016	7,823
Prior	5,919
Revolving	0
Revolving Converted to Term During the Period	0
Total	65,776
Business loans secured by real estate | Franchise real estate secured | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	2,911
2019	49,851
2018	25,927
2017	9,859
2016	11,062
Prior	14,848
Revolving	0
Revolving Converted to Term During the Period	0
Total	114,458
Business loans secured by real estate | Franchise real estate secured | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	9,106
2019	11,697
2018	22,554
2017	48,822
2016	1,126
Prior	2,122
Revolving	0
Revolving Converted to Term During the Period	0
Total	95,427
Retail loans
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	4,935
2019	9,600
2018	16,423
2017	53,030
2016	38,397
Prior	148,142
Revolving	40,952
Revolving Converted to Term During the Period	0
Total	311,479	305,999
Retail loans | Single family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	265,170	255,024
Retail loans | Single family residential | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	4,863
2019	8,236
2018	13,274
2017	9,719
2016	31,203
Prior	96,475
Revolving	27,050
Revolving Converted to Term During the Period	0
Total	190,820
Retail loans | Single family residential | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	1,190
2018	2,268
2017	4,794
2016	2,660
Prior	18,190
Revolving	9,095
Revolving Converted to Term During the Period	0
Total	38,197
Retail loans | Single family residential | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	466
2016	3,178
Prior	9,030
Revolving	959
Revolving Converted to Term During the Period	0
Total	13,633
Retail loans | Single family residential | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	11
2017	12
2016	1,333
Prior	21,128
Revolving	36
Revolving Converted to Term During the Period	0
Total	22,520
Retail loans | Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	46,309	$ 50,975
Retail loans | Consumer | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	72
2019	95
2018	863
2017	51
2016	20
Prior	2,661
Revolving	1,993
Revolving Converted to Term During the Period	0
Total	5,755
Retail loans | Consumer | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	59
2018	7
2017	37,988
2016	0
Prior	488
Revolving	1,737
Revolving Converted to Term During the Period	0
Total	40,279
Retail loans | Consumer | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	20
2018	0
2017	0
2016	3
Prior	150
Revolving	51
Revolving Converted to Term During the Period	0
Total	224
Retail loans | Consumer | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	20
Revolving	31
Revolving Converted to Term During the Period	0
Total	$ 51