Residential Loans, at Fair Value - Schedule of Residential Loans at Fair Value (Details) - USD ($) $ in Thousands	Mar. 31, 2023	Dec. 31, 2022
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	$ 3,701,141	$ 3,898,260
(Discount)/premium	(86,897)	(88,739)
Unrealized (losses) gains	(239,388)	(284,441)
Carrying value	3,374,856	3,525,080
Consolidated SLST | VIE, Primary Beneficiary
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	941,329	955,579
(Discount)/premium	(5,801)	(5,815)
Unrealized (losses) gains	(106,375)	(122,182)
Carrying value	829,153	827,582
Residential loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	992,993	1,152,502
(Discount)/premium	(21,949)	(22,179)
Unrealized (losses) gains	(50,044)	(48,939)
Carrying value	921,000	1,081,384
Residential loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	1,766,819	1,790,179
(Discount)/premium	(59,147)	(60,745)
Unrealized (losses) gains	(82,969)	(113,320)
Carrying value	$ 1,624,703	$ 1,616,114