Loans Held for Investment - Composition of Loan Portfolio (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Jan. 01, 2020	Dec. 31, 2019	Mar. 31, 2019	Dec. 31, 2018
Loans Held for Investment
Loans held for investment	$ 8,754,869		$ 8,722,311
Allowance for credit losses for loans held for investment	(115,422)		(35,698)	$ (37,856)	$ (36,072)
Loans held for investment, net	8,639,447		8,686,613
Loans held for sale, at lower of cost or fair value	111		1,672
Unaccreted mark-to-market discount	35,900		40,700
Investor loans secured by real estate
Loans Held for Investment
Loans held for investment	4,105,070		4,153,084
Allowance for credit losses for loans held for investment		$ (9,027)
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied
Loans Held for Investment
Loans held for investment	2,040,198		2,070,141
Allowance for credit losses for loans held for investment	(15,896)		(1,899)	(1,668)	(1,624)
Investor loans secured by real estate | Multifamily
Loans Held for Investment
Loans held for investment	1,625,682		1,575,726
Allowance for credit losses for loans held for investment	(14,722)		(729)	(669)	(740)
Investor loans secured by real estate | Construction and land
Loans Held for Investment
Loans held for investment	377,525		438,786
Allowance for credit losses for loans held for investment	(9,222)		(4,484)	(5,960)	(5,964)
Investor loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	61,665		68,431
Allowance for credit losses for loans held for investment	(935)		(1,915)	(2,704)	(1,827)
Business loans secured by real estate
Loans Held for Investment
Loans held for investment	2,342,700		2,288,175
Allowance for credit losses for loans held for investment		(5,492)
Business loans secured by real estate | SBA secured by real estate
Loans Held for Investment
Loans held for investment	83,640		88,381
Allowance for credit losses for loans held for investment	(4,044)		(2,119)	(1,966)	(1,824)
Business loans secured by real estate | CRE owner-occupied
Loans Held for Investment
Loans held for investment	1,887,632		1,846,554
Allowance for credit losses for loans held for investment	(26,793)		(2,781)	(1,969)	(1,908)
Business loans secured by real estate | Franchise real estate secured
Loans Held for Investment
Loans held for investment	371,428		353,240
Allowance for credit losses for loans held for investment	(7,503)		(592)	(2,173)	(743)
Commercial loans
Loans Held for Investment
Loans held for investment	2,023,027		1,975,053
Allowance for credit losses for loans held for investment		(20,118)
Commercial loans | Commercial and industrial
Loans Held for Investment
Loans held for investment	1,458,969		1,393,270
Allowance for credit losses for loans held for investment	(15,742)		(13,857)	(13,587)	(13,695)
Commercial loans | Franchise non-real estate secured
Loans Held for Investment
Loans held for investment	547,793		564,357
Allowance for credit losses for loans held for investment	(16,616)		(5,816)	(5,698)	(6,066)
Commercial loans | SBA non-real estate secured
Loans Held for Investment
Loans held for investment	16,265		17,426
Allowance for credit losses for loans held for investment	(516)		(445)	(503)	(654)
Retail loans
Loans Held for Investment
Loans held for investment	284,072		305,999
Allowance for credit losses for loans held for investment		$ (1,061)
Retail loans | Single family residential
Loans Held for Investment
Loans held for investment	237,180		255,024
Allowance for credit losses for loans held for investment	(1,137)		(655)	(758)	(808)
Retail loans | Consumer
Loans Held for Investment
Loans held for investment	46,892		50,975
Allowance for credit losses for loans held for investment	$ (2,296)		$ (406)	$ (201)	$ (219)