Summary of Significant Accounting Policies - Narrative (Details)	12 Months Ended
	Dec. 31, 2021 USD ($) store loan	Dec. 31, 2020 USD ($) loan	Jan. 01, 2020 USD ($)	Dec. 31, 2019 USD ($)
Property, Plant and Equipment [Line Items]
Number of loans modified in TDR | loan	0	2
Net loans	$ 6,990,633,000	$ 7,650,633,000
Intangible assets	0	0
Unamortized multifamily FHA MSR's	$ 1,500,000	807,000
Reasonable and Supportable period (in years)	18 months
Historical loss rate period	12 months
Less: allowance for credit losses	$ (74,965,000)	(109,579,000)	$ (73,658,000)	$ (73,658,000)
Liabilities: Reserve for Unfunded Commitments	$ (4,379,000)	$ (5,498,000)	$ 0
Minimum
Property, Plant and Equipment [Line Items]
Days from commitment to closing	30 days
Maximum
Property, Plant and Equipment [Line Items]
Days from commitment to closing	90 days
Furniture Fixtures and Equipment | Minimum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	3 years
Furniture Fixtures and Equipment | Maximum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	7 years
Computer Software and Hardware | Minimum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	3 years
Computer Software and Hardware | Maximum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	5 years
Building Improvements | Minimum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	5 years
Building Improvements | Maximum
Property, Plant and Equipment [Line Items]
Property, plant and equipment, useful life	20 years
Banking Services
Property, Plant and Equipment [Line Items]
Number of stores | store	17
Lending Services
Property, Plant and Equipment [Line Items]
Number of stores | store	5