Other Assets and Other Liabilities - Other Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2023	Dec. 31, 2022
Offsetting [Abstract]
Dividends and dividend equivalents payable	$ 32,151	$ 49,996
Accrued interest payable	23,653	10,629
Accrued expenses and other liabilities in consolidated multi-family properties	21,797	10,511
Accrued expenses	11,515	15,576
Operating lease liabilities	7,102	8,383
Unfunded commitments for residential and multi-family investments	6,587	2,950
Deferred revenue	5,469	7,131
Advanced remittances from residential loan servicers	4,332	9,098
Deferred tax liabilities	2,012	394
Other	3,398	1,323
Total	$ 118,016	$ 115,991
Operating Lease, Liability, Statement of Financial Position [Extensible List]	Total	Total