Fair Value of Financial Instruments - Changes in Unrealized Gains (Losses) Included in Earnings for Level 3 Assets and Liabilities (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2021	Sep. 30, 2020	Sep. 30, 2021	Sep. 30, 2020
Collateralized Debt Obligations | Consolidated SLST | VIE, Primary Beneficiary
Liabilities
Unrealized gains (losses)	$ (1,985)	$ (13,918)	$ 35,905	$ (62,907)
Residential loans | Consolidated SLST | VIE, Primary Beneficiary
Assets
Unrealized gains (losses)	6,287	41,063	(12,585)	28,014
Residential loans | Residential loans
Assets
Unrealized gains (losses)	7,881	35,420	21,476	(883)
Residential loans | Residential Loans Held in Securitization Trusts
Assets
Unrealized gains (losses)	12,791	817	31,015	(823)
Multi-family loans | Preferred Equity and Mezzanine Loan Investments
Assets
Unrealized gains (losses)	538	(955)	1,203	(6,518)
Multi-family loans | Equity investments
Assets
Unrealized gains (losses)	$ (294)	$ 7	$ 634	$ (4,052)