Fair Value of Financial Instruments - Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands		Dec. 31, 2021	Dec. 31, 2020
Financial Assets:
Securities, at fair value	[1]	$ 256,685	$ 399,999
Restricted cash		99,751	7,165
Level 1 | Held at Carrying Value
Financial Assets:
Cash and cash equivalents		304,696	814,354
Restricted cash		99,751	7,165
Financial Liabilities:
Senior notes		0	100,000
Level 1 | Held at Fair Value
Financial Assets:
Cash and cash equivalents		304,696	814,354
Restricted cash		99,751	7,165
Financial Liabilities:
Senior notes		0	100,031
Level 2 | Held at Carrying Value
Financial Assets:
Securities, at fair value		256,685	399,999
Financial Liabilities:
Securitized debt		2,650,473	1,514,509
Convertible senior notes		226,470	225,177
Level 2 | Held at Fair Value
Financial Assets:
Securities, at fair value		256,685	399,999
Financial Liabilities:
Securitized debt		2,646,203	1,519,567
Convertible senior notes		239,292	228,287
Residential whole loans, at fair value | Level 2 | Held at Carrying Value
Financial Assets:
Residential whole loans		1,082,765	0
Residential whole loans, at fair value | Level 2 | Held at Fair Value
Financial Assets:
Residential whole loans		1,082,765	0
Residential whole loans, at fair value | Level 3 | Held at Carrying Value
Financial Assets:
Residential whole loans		6,830,235	5,325,401
Residential whole loans, at fair value | Level 3 | Held at Fair Value
Financial Assets:
Residential whole loans		6,983,686	5,499,303
Agreements with non-mark-to-market collateral provisions | Level 3 | Held at Carrying Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		939,540	1,159,213
Agreements with non-mark-to-market collateral provisions | Level 3 | Held at Fair Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		940,257	1,159,213
Agreements with mark-to-market collateral provisions | Level 3 | Held at Carrying Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		2,403,151	1,124,162
Agreements with mark-to-market collateral provisions | Level 3 | Held at Fair Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		2,403,724	1,124,162
Financing agreements with mark-to-market collateral provisions | Level 2 | Held at Carrying Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		159,148	213,915
Financing agreements with mark-to-market collateral provisions | Level 2 | Held at Fair Value
Financial Liabilities:
Financing agreements with mark-to-market collateral provisions		$ 159,148	$ 213,915

[1] See Note 6 for information regarding the Company’s pledged assets.