Loans - Net Loans Held-for-Investment (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019
Loans and Leases Receivable Disclosure [Line Items]
Loans held-for-investment, net	$ 3,817,273		$ 3,823,238
Less: allowance for loan losses	(39,493)	$ (43,197)	(37,607)	$ (38,520)	$ (36,800)	$ (28,707)
Net loans held-for-investment	3,777,780		3,785,631
Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Deferred origination loan costs, net	0		4,795
Loans held-for-investment, net	3,800,581		3,804,720
Less: allowance for loan losses	(34,673)	(37,915)	(36,726)	(37,731)	(36,011)	(27,918)
PCD/PCI loans
Loans and Leases Receivable Disclosure [Line Items]
Loans held-for-investment, net	16,692		18,518
Less: allowance for loan losses	(4,820)	(5,282)	(881)	(789)	(789)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	3,588,831		3,602,544
Loans held-for-investment, net	3,588,831		3,610,210
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Multifamily
Loans and Leases Receivable Disclosure [Line Items]
Loans	2,496,613		2,509,310
Loans held-for-investment, net	2,496,613		2,512,934
Less: allowance for loan losses				(27,243)	(26,889)	(20,203)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Commercial mortgage
Loans and Leases Receivable Disclosure [Line Items]
Loans	735,303		716,973
Loans held-for-investment, net	735,303		717,566
Less: allowance for loan losses				(6,699)	(5,444)	(4,891)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | One-to-four family residential mortgage
Loans and Leases Receivable Disclosure [Line Items]
Loans	193,976		210,817
Loans held-for-investment, net	193,976		212,222
Less: allowance for loan losses	(4,610)	(5,172)	(207)	(272)	(312)	(180)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Home equity and lines of credit
Loans and Leases Receivable Disclosure [Line Items]
Loans	99,814		91,126
Loans held-for-investment, net	99,814		93,137
Less: allowance for loan losses	(663)	(643)	(260)	(558)	(574)	(317)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Construction and land
Loans and Leases Receivable Disclosure [Line Items]
Loans	63,125		74,318
Loans held-for-investment, net	63,125		74,351
Less: allowance for loan losses	(259)	(269)	(1,214)	(937)	(671)	(536)
Commercial and industrial and other loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	211,750		197,381
Commercial and industrial loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	209,580		194,352
Loans held-for-investment, net	209,580		191,481
Less: allowance for loan losses	(2,594)	(2,940)	(1,842)	(1,846)	(2,035)	(1,640)
Other loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	2,170		3,029
Loans held-for-investment, net	2,170		3,029
Less: allowance for loan losses	(7)	$ (9)	(198)	$ (176)	$ (86)	$ (151)
Paycheck Protection Program | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	$ 132,700		$ 126,500