Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Mar. 31, 2022	Dec. 31, 2021	Mar. 31, 2021	Dec. 31, 2020
Loans Held for Investment
Loans held for investment	$ 14,733,755	$ 14,295,897
Allowance for credit losses for loans held for investment	(197,517)	(197,752)	$ (266,999)	$ (268,018)
Loans held for investment, net	14,536,238	14,098,145
Loans held for sale, at lower of cost or fair value	11,646	10,869
Net deferred origination fees (costs)	3,900	3,500
Unaccreted mark-to-market discount	71,200	77,100
Total unfunded loan commitments
Loans Held for Investment
Loans held for investment	2,940,370	2,507,911
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	9,162,188	8,987,628
Investor loans secured by real estate | CRE non-owner-occupied
Loans Held for Investment
Loans held for investment	2,774,650	2,771,137
Allowance for credit losses for loans held for investment	(35,974)	(37,380)	(45,545)	(49,176)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	6,041,085	5,891,934
Allowance for credit losses for loans held for investment	(54,325)	(55,209)	(79,815)	(62,534)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	303,811	277,640
Allowance for credit losses for loans held for investment	(5,219)	(5,211)	(13,263)	(12,435)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	42,642	46,917
Allowance for credit losses for loans held for investment	(3,050)	(3,201)	(5,141)	(5,159)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,844,717	2,700,579
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	68,466	69,184
Allowance for credit losses for loans held for investment	(5,195)	(4,866)	(6,451)	(6,567)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	2,391,984	2,251,014
Allowance for credit losses for loans held for investment	(31,891)	(29,575)	(41,594)	(50,517)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	384,267	380,381
Allowance for credit losses for loans held for investment	(7,977)	(7,985)	(10,876)	(11,451)
Commercial loans
Loans Held for Investment
Loans held for investment	2,641,715	2,506,733
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	2,242,632	2,103,112
Allowance for credit losses for loans held for investment	(38,598)	(38,136)	(43,373)	(46,964)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	388,322	392,576
Allowance for credit losses for loans held for investment	(14,304)	(15,084)	(18,903)	(20,525)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	10,761	11,045
Allowance for credit losses for loans held for investment	(490)	(565)	(890)	(995)
Retail loans
Loans Held for Investment
Loans held for investment	85,135	100,957
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	79,978	95,292
Allowance for credit losses for loans held for investment	(233)	(255)	(822)	(1,204)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	5,157	5,665
Allowance for credit losses for loans held for investment	$ (261)	$ (285)	$ (326)	$ (491)