Fair Value of Financial Instruments (Level 3 Assets) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Changes in unrealized gain/losses			$ 236	$ 0
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value (1)
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	$ 1,555,619	$ 775,152	1,325,115	814,682
Purchases and capitalized advances	6,175	4,831	317,300	10,164
Changes in fair value recorded in Net gain on residential whole loans held at fair value	4,599	4,262	18,346	7,209
Collection of principal, net of liquidation gains/losses	(54,184)	(15,652)	(100,868)	(35,695)
Repurchases	(867)	(450)	(1,061)	(756)
Transfers	(42,802)	(24,071)	(90,292)	(51,532)
Balance at end of period	1,468,540	744,072	1,468,540	744,072
Recurring basis | Level 3 | Term Notes Backed by MSR Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	332,040	282,332	381,804	0
Purchases and capitalized advances	49,350	0	149,350	150,000
Collection of principal, net of liquidation gains/losses	0	(8,371)	(150,000)	(17,019)
Changes in unrealized gain/losses	0	0
Transfers from Level 2 to Level 3 (1)	0	0
Transfers			0	140,980
Balance at end of period	381,390	273,961	381,390	273,961
Residential whole loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Excluded amount, unsettled purchases, residential whole loans, at fair value	$ 34,400	$ 239,200	$ 34,400	$ 239,200