Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	Dec. 31, 2022	Dec. 31, 2021
Assets:
Total assets carried at fair value	$ 6,060,888	$ 5,562,034
Liabilities:
Securitized debt	2,435,370	1,316,131
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	5,727,524	5,305,349
Liabilities:
Total liabilities carried at fair value	3,898,744	3,266,773
Securities, at fair value
Assets:
Securities, at fair value	333,364	256,685
Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	578,879	1,322,362
Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	884,495	628,280
Level 1
Assets:
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 1 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Total assets carried at fair value	384,458	1,339,450
Liabilities:
Securitized debt	2,435,370	1,316,131
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	51,094	1,082,765
Liabilities:
Total liabilities carried at fair value	2,435,370	1,316,131
Level 2 | Securities, at fair value
Assets:
Securities, at fair value	333,364	256,685
Level 2 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 3
Assets:
Total assets carried at fair value	5,676,430	4,222,584
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	5,676,430	4,222,584
Liabilities:
Total liabilities carried at fair value	1,463,374	1,950,642
Level 3 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 3 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	578,879	1,322,362
Level 3 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	$ 884,495	$ 628,280