Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2022	Dec. 31, 2021
Assets:
Total assets carried at fair value	$ 6,227,486	$ 5,562,034
Liabilities:
Securitized debt	1,685,796	1,316,131
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	5,977,315	5,305,349
Liabilities:
Total liabilities carried at fair value	3,804,906	3,266,773
Securities, at fair value
Assets:
Securities, at fair value	250,171	256,685
Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	563,860	1,322,362
Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	1,555,250	628,280
Level 1
Assets:
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 1 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Total assets carried at fair value	1,241,804	1,339,450
Liabilities:
Securitized debt	1,685,796	1,316,131
Transfer to Level 2	598,700
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	991,633	1,082,765
Liabilities:
Total liabilities carried at fair value	1,685,796	1,316,131
Level 2 | Securities, at fair value
Assets:
Securities, at fair value	250,171	256,685
Level 2 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 3
Assets:
Total assets carried at fair value	4,985,682	4,222,584
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	4,985,682	4,222,584
Liabilities:
Total liabilities carried at fair value	2,119,110	1,950,642
Level 3 | Securities, at fair value
Assets:
Securities, at fair value	0	0
Level 3 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	563,860	1,322,362
Level 3 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value	$ 1,555,250	$ 628,280