Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019	Mar. 31, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Residential whole loans held at fair value for which the closing price of the purchase transaction had not occurred		$ 87,000		$ 87,000	$ 70,600
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	$ 1,243,792	1,512,337	$ 1,381,583	1,471,263
Purchases and capitalized advances	4,679	5,299	8,198	135,388
Changes in fair value recorded in Net gain on residential whole loans measured at fair value through earnings	2,010	21,188	(72,546)	20,128
Collection of principal, net of liquidation gains/(losses)	(28,608)	(43,072)	(52,412)	(74,823)
Sales and repurchases	(18,225)	(898)	(18,530)	(1,216)
Transfer to REO	(2,667)	(56,027)	(45,312)	(111,913)
Balance at end of period	$ 1,200,981	1,438,827	$ 1,200,981	1,438,827
Recurring basis | Level 3 | Term Notes Backed by MSR-Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period		753,594		538,499
Purchases and capitalized advances		353,970		573,136
Collection of principal, net of liquidation gains/(losses)		(4,392)		(8,976)
Changes in unrealized gain/(losses)		2,854		3,367
Balance at end of period		$ 1,106,026		$ 1,106,026