Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Unsettled residential whole loans	$ 112,200
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	1,216,902	$ 1,381,583
Purchases and capitalized advances	0	0
Changes in fair value recorded in Net gain on residential whole loans measured at fair value through earnings	32,088	(74,556)
Repayments	(25,571)	(20,285)
Sales and repurchases	0	(305)
Transfer to REO	(15,422)	(42,645)
Balance at end of period	$ 1,207,997	$ 1,243,792