Fair Value of Financial Instruments - Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands		Sep. 30, 2023	Dec. 31, 2022
Financial Assets:
Securities, at fair value	[1]	$ 723,959	$ 333,364
Restricted cash		153,449	159,898
Carrying Value | Level 2
Financial Assets:
Securities, at fair value		723,959	333,364
Financial Liabilities:
Securitized debt		4,332,936	3,357,590
Convertible senior notes		218,596	227,845
Carrying Value | Level 1
Financial Assets:
Cash and cash equivalents		300,089	334,183
Restricted cash		153,449	159,898
Estimated Fair Value | Level 2
Financial Assets:
Securities, at fair value		723,959	333,364
Financial Liabilities:
Securitized debt		4,210,440	3,217,905
Convertible senior notes		219,589	211,015
Estimated Fair Value | Level 1
Financial Assets:
Cash and cash equivalents		300,089	334,183
Restricted cash		153,449	159,898
Residential whole loans, at fair value | Carrying Value | Level 3
Financial Assets:
Residential whole loans		8,371,068	7,467,645
Residential whole loans, at fair value | Carrying Value | Level 2
Financial Assets:
Residential whole loans		53,148	51,094
Residential whole loans, at fair value | Estimated Fair Value | Level 3
Financial Assets:
Residential whole loans		8,285,799	7,397,421
Residential whole loans, at fair value | Estimated Fair Value | Level 2
Financial Assets:
Residential whole loans		53,148	51,094
Financing agreements with non-mark-to-market collateral provisions | Carrying Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,124,953	1,003,604
Financing agreements with non-mark-to-market collateral provisions | Estimated Fair Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,125,773	1,004,260
Agreements with mark-to-market collateral provisions | Carrying Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,752,492	2,111,396
Agreements with mark-to-market collateral provisions | Estimated Fair Value | Level 3
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		1,753,355	2,111,647
Financing agreements with mark-to-market collateral provisions | Carrying Value | Level 2
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		608,996	111,651
Financing agreements with mark-to-market collateral provisions | Estimated Fair Value | Level 2
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		$ 608,996	$ 111,651

[1] See Note 6 for information regarding the Company’s pledged assets.