Allowance for Credit Losses - Amortized Cost of Loans by Estimated LTV and FICO Bands (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	$ 1,212,344
2019	3,540,465
2018	2,369,668
2017	1,982,912
2016	1,171,944
Prior	2,385,545
Revolving	779,125
Revolving Converted to Term During the Period	8,837
Total	13,450,840	$ 8,722,311
Investor loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	854,784
2019	2,458,374
2018	1,560,916
2017	1,114,295
2016	730,632
Prior	1,513,942
Revolving	12,538
Revolving Converted to Term During the Period	0
Total	8,245,481	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,707,930	2,070,141
Investor loans secured by real estate | CRE non-owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	100,859
2019	238,182
2018	190,263
2017	156,791
2016	192,044
Prior	602,788
Revolving	11,031
Revolving Converted to Term During the Period	0
Total	1,491,958
Investor loans secured by real estate | CRE non-owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	71,824
2019	220,089
2018	110,830
2017	136,117
2016	84,644
Prior	217,812
Revolving	559
Revolving Converted to Term During the Period	0
Total	841,875
Investor loans secured by real estate | CRE non-owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	18,018
2019	105,825
2018	171,532
2017	18,946
2016	13,796
Prior	30,887
Revolving	0
Revolving Converted to Term During the Period	0
Total	359,004
Investor loans secured by real estate | CRE non-owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	2,759
2018	4,779
2017	2,771
2016	212
Prior	4,572
Revolving	0
Revolving Converted to Term During the Period	0
Total	15,093
Investor loans secured by real estate | Multifamily
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	5,142,069	1,575,726
Investor loans secured by real estate | Multifamily | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	147,694
2019	347,848
2018	303,801
2017	252,662
2016	90,318
Prior	294,211
Revolving	574
Revolving Converted to Term During the Period	0
Total	1,437,108
Investor loans secured by real estate | Multifamily | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	233,564
2019	742,570
2018	406,271
2017	234,044
2016	169,033
Prior	242,603
Revolving	0
Revolving Converted to Term During the Period	0
Total	2,028,085
Investor loans secured by real estate | Multifamily | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	257,091
2019	628,393
2018	244,007
2017	262,512
2016	153,542
Prior	96,542
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,642,087
Investor loans secured by real estate | Multifamily | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	16,424
2018	10,661
2017	1,889
2016	0
Prior	5,815
Revolving	0
Revolving Converted to Term During the Period	0
Total	34,789
Investor loans secured by real estate | Construction and land
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	337,872	438,786
Investor loans secured by real estate | Construction and land | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	24,116
2019	129,160
2018	66,830
2017	26,261
2016	19,932
Prior	7,997
Revolving	374
Revolving Converted to Term During the Period	0
Total	274,670
Investor loans secured by real estate | Construction and land | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	1,124
2019	13,254
2018	23,699
2017	6,762
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	44,839
Investor loans secured by real estate | Construction and land | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	3,295
2018	15,068
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	18,363
Investor loans secured by real estate | Construction and land | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	57,610	68,431
Investor loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	2,070
2018	653
2017	673
2016	330
Prior	785
Revolving	0
Revolving Converted to Term During the Period	0
Total	4,511
Investor loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	2,433
2018	1,643
2017	4,017
2016	621
Prior	4,482
Revolving	0
Revolving Converted to Term During the Period	0
Total	13,196
Investor loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	3,905
2018	5,075
2017	4,185
2016	4,795
Prior	1,897
Revolving	0
Revolving Converted to Term During the Period	0
Total	19,857
Investor loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	494
2019	2,167
2018	5,804
2017	6,665
2016	1,365
Prior	3,551
Revolving	0
Revolving Converted to Term During the Period	0
Total	20,046
Business loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	233,812
2019	523,420
2018	450,328
2017	479,759
2016	283,961
Prior	587,025
Revolving	4,938
Revolving Converted to Term During the Period	0
Total	2,563,243	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	84,126	88,381
Business loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	1,355
2019	1,633
2018	5,376
2017	5,683
2016	3,175
Prior	15,281
Revolving	95
Revolving Converted to Term During the Period	0
Total	32,598
Business loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	104
2019	513
2018	1,802
2017	1,719
2016	3,706
Prior	5,665
Revolving	0
Revolving Converted to Term During the Period	0
Total	13,509
Business loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	264
2019	3,148
2018	751
2017	4,193
2016	2,340
Prior	5,401
Revolving	0
Revolving Converted to Term During the Period	0
Total	16,097
Business loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	920
2019	2,364
2018	6,147
2017	7,509
2016	1,336
Prior	3,646
Revolving	0
Revolving Converted to Term During the Period	0
Total	21,922
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,119,788	1,846,554
Business loans secured by real estate | CRE owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	56,954
2019	156,880
2018	171,740
2017	203,875
2016	133,740
Prior	357,734
Revolving	4,843
Revolving Converted to Term During the Period	0
Total	1,085,766
Business loans secured by real estate | CRE owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	56,139
2019	93,895
2018	97,245
2017	94,359
2016	73,389
Prior	80,933
Revolving	0
Revolving Converted to Term During the Period	0
Total	495,960
Business loans secured by real estate | CRE owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	55,932
2019	155,872
2018	80,898
2017	46,241
2016	32,547
Prior	50,458
Revolving	0
Revolving Converted to Term During the Period	0
Total	421,948
Business loans secured by real estate | CRE owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	41,637
2019	21,366
2018	11,910
2017	13,424
2016	2,216
Prior	25,561
Revolving	0
Revolving Converted to Term During the Period	0
Total	116,114
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	359,329	353,240
Business loans secured by real estate | Franchise real estate secured | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	7,462
2019	13,322
2018	14,407
2017	21,162
2016	11,592
Prior	20,549
Revolving	0
Revolving Converted to Term During the Period	0
Total	88,494
Business loans secured by real estate | Franchise real estate secured | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	9,981
2018	15,893
2017	23,658
2016	7,784
Prior	5,862
Revolving	0
Revolving Converted to Term During the Period	0
Total	63,178
Business loans secured by real estate | Franchise real estate secured | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	3,972
2019	53,584
2018	21,750
2017	9,768
2016	11,017
Prior	14,697
Revolving	0
Revolving Converted to Term During the Period	0
Total	114,788
Business loans secured by real estate | Franchise real estate secured | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	9,073
2019	10,862
2018	22,409
2017	48,168
2016	1,119
Prior	1,238
Revolving	0
Revolving Converted to Term During the Period	0
Total	92,869
Retail loans
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	4,532
2019	8,527
2018	15,032
2017	52,921
2016	35,086
Prior	137,421
Revolving	34,874
Revolving Converted to Term During the Period	0
Total	288,393	305,999
Retail loans | Single family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	243,359	255,024
Retail loans | Single family residential | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	4,469
2019	7,198
2018	12,719
2017	9,658
2016	29,291
Prior	91,460
Revolving	22,566
Revolving Converted to Term During the Period	0
Total	177,361
Retail loans | Single family residential | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	1,187
2018	2,253
2017	4,763
2016	2,641
Prior	28,625
Revolving	7,919
Revolving Converted to Term During the Period	0
Total	47,388
Retail loans | Single family residential | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	463
2016	3,142
Prior	11,113
Revolving	932
Revolving Converted to Term During the Period	0
Total	15,650
Retail loans | Single family residential | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	2,925
Revolving	35
Revolving Converted to Term During the Period	0
Total	2,960
Retail loans | Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	45,034	$ 50,975
Retail loans | Consumer | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	63
2019	85
2018	54
2017	46
2016	10
Prior	2,648
Revolving	1,670
Revolving Converted to Term During the Period	0
Total	4,576
Retail loans | Consumer | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	40
2018	6
2017	37,991
2016	0
Prior	480
Revolving	1,665
Revolving Converted to Term During the Period	0
Total	40,182
Retail loans | Consumer | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	17
2018	0
2017	0
2016	2
Prior	144
Revolving	59
Revolving Converted to Term During the Period	0
Total	222
Retail loans | Consumer | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	26
Revolving	28
Revolving Converted to Term During the Period	0
Total	$ 54