Financing Agreements - Pledged Collateral (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	$ 8,019,405	$ 5,799,786
Accrued interest on assets	25,700	24,600
Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	1,426,051	1,927,464
Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	3,556,273	2,004,180
Securitized
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	3,037,081	1,868,142
Residential whole loans, at carrying value
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	2,629,919	4,140,961
Residential whole loans, at carrying value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	693,982	1,497,281
Residential whole loans, at carrying value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	459,349	1,207,364
Residential whole loans, at carrying value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	1,476,588	1,436,316
Residential whole loans, at fair value
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	5,042,356	1,209,349
Residential whole loans, at fair value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	706,377	430,183
Residential whole loans, at fair value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	2,810,865	396,817
Residential whole loans, at fair value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	1,525,114	382,349
Securities, at fair value
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	256,685	399,999
Securities, at fair value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	0	0
Securities, at fair value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	256,685	399,999
Securities, at fair value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	0	0
Other assets: REO
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	90,445	49,477
Other assets: REO | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	25,692	0
Other assets: REO | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	29,374	0
Other assets: REO | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged against the repurchase agreements	$ 35,379	$ 49,477