Financing Agreements (Pledged Collateral) (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Dec. 31, 2020
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	$ 7,024,171	$ 5,842,560
Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,369,924	1,927,464
Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	3,287,985	2,046,954
Securitized
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	2,366,262	1,868,142
Residential whole loans, at carrying value
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	3,007,602	4,140,961
Residential whole loans, at carrying value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	794,267	1,497,281
Residential whole loans, at carrying value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	533,633	1,207,364
Residential whole loans, at carrying value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,679,702	1,436,316
Residential whole loans, at fair value
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	3,705,529	1,209,349
Residential whole loans, at fair value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	555,738	430,183
Residential whole loans, at fair value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	2,499,284	396,817
Residential whole loans, at fair value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	650,507	382,349
Securities, at fair value
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	222,705	442,773
Securities, at fair value | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Securities, at fair value | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	222,705	442,773
Securities, at fair value | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Other assets: REO
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	88,335	49,477
Other assets: REO | Non-Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	19,919	0
Other assets: REO | Mark-to-Market
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	32,363	0
Other assets: REO | Securitized
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	$ 36,053	$ 49,477