Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	9 Months Ended
	Sep. 30, 2022	Dec. 31, 2021
Assets:
Total assets carried at fair value	$ 6,543,373	$ 5,562,034
Liabilities:
Securitized debt	2,860,253	1,316,131
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	6,315,966	5,305,349
Liabilities:
Total liabilities carried at fair value	4,397,470	3,266,773
Securities, at fair value
Assets:
Securities, at fair value	227,407	256,685
Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	604,721	1,322,362
Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	932,496	628,280
Level 1
Assets:
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 1 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Total assets carried at fair value	1,080,403	1,339,450
Liabilities:
Securitized debt	2,860,253	1,316,131
Transfer to Level 2	508,600
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	852,996	1,082,765
Liabilities:
Total liabilities carried at fair value	2,860,253	1,316,131
Level 2 | Securities, at fair value
Assets:
Securities, at fair value	227,407	256,685
Level 2 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 3
Assets:
Total assets carried at fair value	5,462,970	4,222,584
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	5,462,970	4,222,584
Liabilities:
Total liabilities carried at fair value	1,537,217	1,950,642
Level 3 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 3 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	604,721	1,322,362
Level 3 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	$ 932,496	$ 628,280