Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Residential whole loans held at fair value for which the closing price of the purchase transaction had not occurred		$ 70,600
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	$ 1,381,583	1,471,263
Purchases and capitalized advances	3,520	130,089
Changes in fair value recorded in Net gain on residential whole loans measured at fair value through earnings	(74,556)	(1,060)
Collection of principal, net of liquidation gains/(losses)	(23,805)	(31,751)
Repurchases	(305)	(318)
Transfer to REO	(42,645)	(55,886)
Balance at end of period	$ 1,243,792	1,512,337
Recurring basis | Level 3 | Term Notes Backed by MSR-Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period		538,499
Purchases and capitalized advances		219,166
Collection of principal, net of liquidation gains/(losses)		(4,584)
Changes in unrealized gain/(losses)		513
Transfer to Level 3		0
Balance at end of period		$ 753,594