Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	Mar. 31, 2024	Dec. 31, 2023
Assets:
Total assets carried at fair value	$ 8,390,857	$ 8,257,598
Liabilities:
Securitized debt	4,065,630	3,985,372
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	7,653,907	7,511,508
Liabilities:
Total liabilities carried at fair value	4,641,438	4,633,660
Securities, at fair value
Assets:
Securities, at fair value	736,950	746,090
Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	399,049	469,424
Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	176,759	178,864
Level 1
Assets:
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 1 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Total assets carried at fair value	791,604	801,869
Liabilities:
Securitized debt	4,065,630	3,985,372
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	54,654	55,779
Liabilities:
Total liabilities carried at fair value	4,065,630	3,985,372
Level 2 | Securities, at fair value
Assets:
Securities, at fair value	736,950	746,090
Level 2 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 3
Assets:
Total assets carried at fair value	7,599,253	7,455,729
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	7,599,253	7,455,729
Liabilities:
Total liabilities carried at fair value	575,808	648,288
Level 3 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 3 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	399,049	469,424
Level 3 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	$ 176,759	$ 178,864