Fair Value of Financial Instruments - Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands		Sep. 30, 2017	Jun. 30, 2017	Dec. 31, 2016	Sep. 30, 2016
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	[1]	$ 1,103,518		$ 814,682
Total Residential Whole Loans, Unpaid Principal Balance		1,178,866		966,174
Residential whole loans held at fair value excluded, purchase transaction not occurred		120,400	$ 92,700		$ 92,800
Residential whole loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value		983,150		814,682
Total Residential Whole Loans, Unpaid Principal Balance		1,178,866		966,174
Total Residential Whole Loans, Difference		(195,716)		(151,492)
Residential Whole Loans 90 days or more past due, Fair Value		690,924		570,025
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance		853,655		695,282
Residential Whole Loans 90 days or more past due, Difference		$ (162,731)		$ (125,257)

[1] Includes approximately $131.3 million of Residential whole loans, at carrying value and $40.4 million of Residential whole loans, at fair value transferred to a consolidated VIE at September 30, 2017. Such assets can be used only to settle the obligations of the VIE.