Fair Value of Financial Instruments - Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands		Mar. 31, 2022	Dec. 31, 2021
Financial Assets:
Securities, at fair value	[1]	$ 250,171	$ 256,685
Restricted cash		144,600	99,751
Carrying Value | Level 1
Financial Assets:
Cash and cash equivalents		410,939	304,696
Restricted cash		144,600	99,751
Carrying Value | Level 2
Financial Assets:
Securities, at fair value		250,171	256,685
Financial Liabilities:
Securitized debt		2,859,061	2,650,473
Convertible senior notes		226,807	226,470
Estimated Fair Value | Level 1
Financial Assets:
Cash and cash equivalents		410,939	304,696
Restricted cash		144,600	99,751
Estimated Fair Value | Level 2
Financial Assets:
Securities, at fair value		250,171	256,685
Financial Liabilities:
Securitized debt		2,811,392	2,646,203
Convertible senior notes		231,758	239,292
Residential whole loans, at fair value | Carrying Value | Level 2
Financial Assets:
Residential whole loans		991,633	1,082,765
Residential whole loans, at fair value | Carrying Value | Level 3
Financial Assets:
Residential whole loans		7,270,272	6,830,235
Residential whole loans, at fair value | Estimated Fair Value | Level 2
Financial Assets:
Residential whole loans		991,633	1,082,765
Residential whole loans, at fair value | Estimated Fair Value | Level 3
Financial Assets:
Residential whole loans		7,324,320	6,983,686
Financing agreements with non-mark-to-market collateral provisions | Carrying Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,001,408	939,540
Financing agreements with non-mark-to-market collateral provisions | Estimated Fair Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,002,234	940,257
Agreements with mark-to-market collateral provisions | Carrying Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		2,781,916	2,403,151
Agreements with mark-to-market collateral provisions | Estimated Fair Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		2,782,240	2,403,724
Financing agreements with mark-to-market collateral provisions | Carrying Value | Level 2
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		159,019	159,148
Financing agreements with mark-to-market collateral provisions | Estimated Fair Value | Level 2
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		$ 159,019	$ 159,148

[1] See Note 6 for information regarding the Company’s pledged assets.