Fair Value of Financial Instruments - Level 3 Assets (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in unrealized gain/losses	$ 804	$ 0
Recurring basis | Level 3 | Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Balance at beginning of period	814,682	623,276
Purchases	683,735	316,407
Changes in unrealized gain/losses	33,617	31,254
Collection of principal	(87,072)	(66,694)
Repurchases	(2,716)	(2,909)
Transfers to REO	(117,131)	(86,652)
Balance at end of period	1,325,115	814,682
Recurring basis | Level 3 | Term Notes Backed by MSR Related Collateral
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Balance at beginning of period	0	0
Purchases	381,000	0
Collection of principal	(140,980)	0
Transfers to REO	140,980	0
Balance at end of period	$ 381,804	$ 0