Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	Dec. 31, 2023	Dec. 31, 2022
Assets:
Total assets carried at fair value	$ 8,257,598	$ 6,060,888
Liabilities:
Securitized debt	3,985,372	2,435,370
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	7,511,508	5,727,524
Liabilities:
Total liabilities carried at fair value	4,633,660	3,898,744
Securities, at fair value
Assets:
Securities, at fair value	746,090	333,364
Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	469,424	578,879
Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	178,864	884,495
Level 1
Assets:
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 1 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Total assets carried at fair value	801,869	384,458
Liabilities:
Securitized debt	3,985,372	2,435,370
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	55,779	51,094
Liabilities:
Total liabilities carried at fair value	3,985,372	2,435,370
Level 2 | Securities, at fair value
Assets:
Securities, at fair value	746,090	333,364
Level 2 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 3
Assets:
Total assets carried at fair value	7,455,729	5,676,430
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	7,455,729	5,676,430
Liabilities:
Total liabilities carried at fair value	648,288	1,463,374
Level 3 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 3 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	469,424	578,879
Level 3 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	$ 178,864	$ 884,495