Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Assets:
Securities, at fair value	$ 256,685	$ 399,999
Total assets carried at fair value	5,562,034	1,616,901
Liabilities:
Securitized debt	1,316,131	869,482
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	5,305,349	1,216,902
Liabilities:
Total liabilities carried at fair value	3,266,773	3,366,772
Securities, at fair value
Assets:
Securities, at fair value	153,800	239,000
Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	628,280	1,159,213
Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	1,322,362	1,338,077
Level 1
Assets:
Securities, at fair value	0	0
Total assets carried at fair value	0	0
Liabilities:
Securitized debt	0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	0	0
Liabilities:
Total liabilities carried at fair value	0	0
Level 1 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 1 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2
Assets:
Securities, at fair value	256,685	399,999
Total assets carried at fair value	1,339,450	399,999
Liabilities:
Securitized debt	1,316,131	869,482
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	1,082,765	0
Liabilities:
Total liabilities carried at fair value	1,316,131	1,083,397
Level 2 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	0
Level 2 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	0	213,915
Level 3
Assets:
Securities, at fair value	0	0
Total assets carried at fair value	4,222,584	1,216,902
Liabilities:
Securitized debt	0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value	4,222,584	1,216,902
Liabilities:
Total liabilities carried at fair value	1,950,642	2,283,375
Level 3 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	628,280	1,159,213
Level 3 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value	$ 1,322,362	$ 1,124,162