Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Loans Held for Investment
Loans held for investment	$ 14,295,897	$ 13,236,433
Allowance for credit losses for loans held for investment	(197,752)	(268,018)	$ (35,698)	$ (36,072)
Loans held for investment, net	14,098,145	12,968,415
Loans held for sale, at lower of cost or fair value	10,869	601
Net deferred origination fees (costs)	3,500	2,600
Unaccreted mark-to-market discount	77,100	113,800
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	2,507,911	1,947,250
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	8,987,628	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,771,137	2,675,085
Allowance for credit losses for loans held for investment	(37,380)	(49,176)	(1,899)	(1,624)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	5,891,934	5,171,356
Allowance for credit losses for loans held for investment	(55,209)	(62,534)	(729)	(740)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	277,640	321,993
Allowance for credit losses for loans held for investment	(5,211)	(12,435)	(4,484)	(5,964)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	46,917	57,331
Allowance for credit losses for loans held for investment	(3,201)	(5,159)	(1,915)	(1,827)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,700,579	2,541,577
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	69,184	79,595
Allowance for credit losses for loans held for investment	(4,866)	(6,567)	(2,119)	(1,824)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,251,014	2,114,050
Allowance for credit losses for loans held for investment	(29,575)	(50,517)	(2,781)	(1,908)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	380,381	347,932
Allowance for credit losses for loans held for investment	(7,985)	(11,451)	(592)	(743)
Commercial loans
Loans Held for Investment
Loans held for investment	2,506,733	2,229,588
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	2,103,112	1,768,834
Allowance for credit losses for loans held for investment	(38,136)	(46,964)	(13,857)	(13,695)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	392,576	444,797
Allowance for credit losses for loans held for investment	(15,084)	(20,525)	(5,816)	(6,066)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	11,045	15,957
Allowance for credit losses for loans held for investment	(565)	(995)	(445)	(654)
Retail loans
Loans Held for Investment
Loans held for investment	100,957	239,503
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	95,292	232,574
Allowance for credit losses for loans held for investment	(255)	(1,204)	(655)	(808)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	5,665	6,929
Allowance for credit losses for loans held for investment	$ (285)	$ (491)	$ (406)	$ (219)