Financing Agreements - Pledged Collateral (Details) - USD ($) $ in Thousands	Mar. 31, 2023	Dec. 31, 2022
Debt Instrument [Line Items]
Accrued interest on assets	$ 27,200	$ 30,900
Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	8,350,163	8,057,175
Non-Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,251,617	1,331,386
Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	2,401,335	2,654,442
Securitized | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	4,697,211	4,071,347
Residential whole loans, at carrying value | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,744,490	1,814,780
Residential whole loans, at carrying value | Non-Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	206,613	215,993
Residential whole loans, at carrying value | Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	252,288	284,683
Residential whole loans, at carrying value | Securitized | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,285,589	1,314,104
Residential whole loans, at fair value | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	6,039,665	5,980,471
Residential whole loans, at fair value | Non-Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,026,326	1,095,556
Residential whole loans, at fair value | Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,635,932	2,164,158
Residential whole loans, at fair value | Securitized | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	3,377,407	2,720,757
Securities, at fair value | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	484,646	177,111
Securities, at fair value | Non-Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Securities, at fair value | Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	484,646	177,111
Securities, at fair value | Securitized | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Other assets: REO | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	81,362	84,813
Other assets: REO | Non-Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	18,678	19,837
Other assets: REO | Mark-to-Market | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	28,469	28,490
Other assets: REO | Securitized | Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	$ 34,215	$ 36,486