Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Residential whole loans held at fair value for which the closing price of the purchase transaction had not occurred	$ 70,600
Recurring basis | Level 3 | Residential whole loans, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	1,381,583	$ 1,471,263
Purchases	0	210,031
Changes in unrealized gains	17,204	47,849
Collection of principal	(92,733)	(127,063)
Sales and repurchases	(18,530)	(1,338)
Transfer to REO	(70,622)	(219,159)
Balance at end of period	1,216,902	1,381,583
Recurring basis | Level 3 | Term Notes Backed by MSR-Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	0	538,499
Purchases	0	573,137
Collection of principal	0	(12,897)
Changes in unrealized gains	0	5,391
Transfer to Level 2	0	(1,104,130)
Balance at end of period	$ 0	$ 0