Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Residential whole loans held at fair value for which the closing price of the purchase transaction had not occurred	$ 70,600	$ 34,400	$ 70,600	$ 34,400
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	1,512,337	1,555,619	1,471,263	1,325,115
Purchases and capitalized advances	5,299	6,175	135,388	317,300
Changes in fair value recorded in Net gain on residential whole loans measured at fair value through earnings	21,188	4,599	20,128	18,346
Collection of principal, net of liquidation gains/losses	(43,072)	(54,184)	(74,823)	(100,868)
Repurchases	(898)	(867)	(1,216)	(1,061)
Transfer to REO	(56,027)	(42,802)	(111,913)	(90,292)
Balance at end of period	1,438,827	1,468,540	1,438,827	1,468,540
Recurring basis | Level 3 | Term Notes Backed by MSR Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	753,594	332,040	538,499	381,804
Purchases and capitalized advances	353,970	49,350	573,136	149,350
Collection of principal, net of liquidation gains/losses	(4,392)	0	(8,976)	(150,000)
Changes in unrealized gain/losses	2,854	0	3,367	236
Balance at end of period	$ 1,106,026	$ 381,390	$ 1,106,026	$ 381,390