Allowance for Credit Losses - Schedule of Allocation of Allowance for Credit Losses (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	$ 195,651	$ 197,752	$ 268,018
Charge-offs	(19,349)	(13,719)	(8,616)
Recoveries	1,746	3,159	5,419
Provision for credit losses	14,423	8,459	(67,069)
Ending ACL Balance	192,471	195,651	197,752
Investor loans secured by real estate
Allocation of allowance as well as the activity in allowance
Charge-offs	(5,452)
Investor loans secured by real estate | CRE non-owner-occupied
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	33,692	37,380	49,176
Charge-offs	(3,472)	(4,760)	(154)
Recoveries	159	0	78
Provision for credit losses	651	1,072	(11,720)
Ending ACL Balance	31,030	33,692	37,380
Investor loans secured by real estate | Multifamily
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	56,334	55,209	62,534
Charge-offs	(1,872)	0	0
Recoveries	1	0	0
Provision for credit losses	1,849	1,125	(7,325)
Ending ACL Balance	56,312	56,334	55,209
Investor loans secured by real estate | Construction and land
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	7,114	5,211	12,435
Charge-offs	0	0	0
Recoveries	0	0	0
Provision for credit losses	2,200	1,903	(7,224)
Ending ACL Balance	9,314	7,114	5,211
Investor loans secured by real estate | SBA secured by real estate
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	2,592	3,201	5,159
Charge-offs	(108)	(70)	(424)
Recoveries	0	0	3
Provision for credit losses	(302)	(539)	(1,537)
Ending ACL Balance	2,182	2,592	3,201
Business loans secured by real estate
Allocation of allowance as well as the activity in allowance
Charge-offs	(2,370)
Business loans secured by real estate | SBA secured by real estate
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	4,348	4,866	6,567
Charge-offs	0	0	(98)
Recoveries	248	0	130
Provision for credit losses	(169)	(518)	(1,733)
Ending ACL Balance	4,427	4,348	4,866
Business loans secured by real estate | CRE owner-occupied
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	32,340	29,575	50,517
Charge-offs	(2,370)	0	0
Recoveries	40	56	56
Provision for credit losses	(1,223)	2,709	(20,998)
Ending ACL Balance	28,787	32,340	29,575
Business loans secured by real estate | Franchise real estate secured
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	7,019	7,985	11,451
Charge-offs	0	0	0
Recoveries	0	0	0
Provision for credit losses	480	(966)	(3,466)
Ending ACL Balance	7,499	7,019	7,985
Commercial loans
Allocation of allowance as well as the activity in allowance
Charge-offs	(10,541)
Commercial loans | Commercial and industrial
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	35,169	38,136	46,964
Charge-offs	(10,474)	(8,387)	(5,384)
Recoveries	1,041	2,904	4,720
Provision for credit losses	10,956	2,516	(8,164)
Ending ACL Balance	36,692	35,169	38,136
Commercial loans | Franchise non-real estate secured
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	16,029	15,084	20,525
Charge-offs	0	(448)	(2,554)
Recoveries	150	0	405
Provision for credit losses	(1,048)	1,393	(3,292)
Ending ACL Balance	15,131	16,029	15,084
Commercial loans | SBA non-real estate secured
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	441	565	995
Charge-offs	(67)	(50)	0
Recoveries	71	51	21
Provision for credit losses	13	(125)	(451)
Ending ACL Balance	458	441	565
Retail loans
Allocation of allowance as well as the activity in allowance
Charge-offs	(986)
Retail loans | Single family residential
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	352	255	1,204
Charge-offs	(90)	0	0
Recoveries	1	148	3
Provision for credit losses	242	(51)	(952)
Ending ACL Balance	505	352	255
Retail loans | Consumer
Allocation of allowance as well as the activity in allowance
Beginning ACL Balance	221	285	491
Charge-offs	(896)	(4)	(2)
Recoveries	35	0	3
Provision for credit losses	774	(60)	(207)
Ending ACL Balance	$ 134	$ 221	$ 285