Residential Loans, at Fair Value - Schedule of Residential Loans at Fair Value (Details) - USD ($) $ in Thousands	Mar. 31, 2022	Dec. 31, 2021
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	$ 4,034,469	$ 3,529,804
(Discount)/premium	(81,161)	(84,265)
Unrealized (losses) gains	119	130,062
Carrying value	3,953,427	3,575,601
Consolidated SLST | VIE, Primary Beneficiary
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	1,037,648	1,071,228
(Discount)/premium	(3,802)	(2,998)
Unrealized (losses) gains	(63,993)	2,652
Carrying value	969,853	1,070,882
Residential loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	1,750,555	1,682,138
(Discount)/premium	(25,427)	(44,256)
Unrealized (losses) gains	(3,910)	65,408
Carrying value	1,721,218	1,703,290
Residential loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal	1,246,266	776,438
(Discount)/premium	(51,932)	(37,011)
Unrealized (losses) gains	68,022	62,002
Carrying value	$ 1,262,356	$ 801,429