Other Assets and Other Liabilities - Other Assets (Details) - USD ($) $ in Thousands	Mar. 31, 2024	Dec. 31, 2023	Mar. 31, 2023
Offsetting [Abstract]
Restricted cash	$ 163,792	$ 143,535	$ 115,249
Real estate owned	41,397	34,353
Accrued interest receivable	39,936	37,312
Other assets in consolidated multi-family properties	24,321	28,923
Recoverable advances on residential loans	18,093	18,328
Other receivables	16,781	12,593
Collections receivable from residential loan servicers	16,732	14,956
Derivative assets	7,143	6,510
Operating lease right-of-use assets	6,308	6,581
Deferred tax assets	4,162	4,510
Lease intangibles, net in consolidated multi-family properties	951	2,378
Other	5,383	5,378
Other assets	$ 344,999	$ 315,357
Operating Lease, Right-of-Use Asset, Statement of Financial Position [Extensible Enumeration]	Other assets	Other assets