Loans - Net Loans Held-for-Investment (Details) - USD ($) $ in Thousands	Sep. 30, 2023	Jun. 30, 2023	Dec. 31, 2022	Sep. 30, 2022	Jun. 30, 2022	Dec. 31, 2021
Loans and Leases Receivable Disclosure [Line Items]
Total loans held-for-investment	$ 4,229,974		$ 4,243,693
Less: allowance for credit losses	(38,480)	$ (41,154)	(42,617)	$ (41,883)	$ (39,031)	$ (38,973)
Net loans held-for-investment	4,191,494		4,201,076
Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Total loans held-for-investment	4,219,603		4,232,191
Less: allowance for credit losses	(35,372)	(37,428)	(38,734)	(37,861)	(34,873)	(34,241)
PCD loans
Loans and Leases Receivable Disclosure [Line Items]
Total loans held-for-investment	10,371		11,502
Less: allowance for credit losses	(3,108)	(3,726)	(3,883)	(4,022)	(4,158)	(4,732)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	4,072,741		4,075,261
Total loans held-for-investment	4,072,741		4,075,261
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Multifamily
Loans and Leases Receivable Disclosure [Line Items]
Loans	2,782,141		2,824,579
Total loans held-for-investment	2,782,141		2,824,579
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Commercial mortgage
Loans and Leases Receivable Disclosure [Line Items]
Loans	932,987		899,249
Total loans held-for-investment	932,987		899,249
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | One-to-four family residential mortgage
Loans and Leases Receivable Disclosure [Line Items]
Loans	164,525		173,946
Total loans held-for-investment	164,525		173,946
Less: allowance for credit losses	(3,571)	(3,814)	(3,936)	(3,565)	(3,005)	(3,545)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Home equity and lines of credit
Loans and Leases Receivable Disclosure [Line Items]
Loans	160,798		152,555
Total loans held-for-investment	160,798		152,555
Less: allowance for credit losses	(1,355)	(1,278)	(866)	(874)	(802)	(560)
Real estate loans: | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration | Construction and land
Loans and Leases Receivable Disclosure [Line Items]
Loans	32,290		24,932
Total loans held-for-investment	32,290		24,932
Less: allowance for credit losses	(299)	(301)	(324)	(275)	(285)	(169)
Commercial and industrial and other loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	146,862		156,930
Commercial and industrial loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	144,788		154,700
Total loans held-for-investment	144,788		154,700
Less: allowance for credit losses	(5,802)	(7,295)	(4,114)	(2,955)	(2,705)	(3,173)
Other loans | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	2,074		2,230
Total loans held-for-investment	2,074		2,230
Less: allowance for credit losses	(7)	$ (7)	(9)	$ (6)	$ (11)	$ (9)
Payment Protection Program | Financial Asset, Other than Financial Asset Acquired with Credit Deterioration
Loans and Leases Receivable Disclosure [Line Items]
Loans	$ 325		$ 5,100