Fair Value of Financial Instruments - Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands		Dec. 31, 2017	Dec. 31, 2016
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	[1]	$ 1,325,115	$ 814,682
Outstanding principal balance		1,562,373	966,174
Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value		1,325,115	814,682
Outstanding principal balance		1,562,373	966,174
Total Residential Whole Loans, Difference		(237,258)	(151,492)
Residential Whole Loans 90 days or more past due, Fair Value		840,572	570,025
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance		1,027,818	695,282
Residential Whole Loans 90 days or more past due, Difference		$ (187,246)	$ (125,257)

[1] Includes approximately $183.2 million of Residential whole loans, at carrying value and $289.3 million of Residential whole loans, at fair value transferred to consolidated VIEs at December 31, 2017. Such assets can be used only to settle the obligations of each respective VIE.