Other Assets and Other Liabilities - Schedule of Other Liabilities (Details) - USD ($) $ in Thousands	Sep. 30, 2024	Dec. 31, 2023
Offsetting [Abstract]
Accrued interest payable	$ 34,633	$ 23,653
Dividends and dividend equivalents payable	30,640	32,151
Advanced remittances from residential loan servicers	19,239	4,332
Accrued expenses and other liabilities in consolidated multi-family properties	14,917	21,797
Unfunded commitments for residential and multi-family investments	10,517	6,587
Accrued expenses	9,227	11,515
Swap margin payable	7,653	0
Operating lease liabilities	6,246	7,102
Deferred revenue	5,560	5,469
Deferred tax liabilities	3,276	2,012
Other	3,886	3,398
Total	$ 145,794	$ 118,016
Operating Lease, Liability, Statement of Financial Position [Extensible Enumeration]	Total	Total