Residential Whole Loans - Narrative (Details)		3 Months Ended		9 Months Ended
		Sep. 30, 2017 USD ($) loan	Sep. 30, 2016 USD ($)	Sep. 30, 2017 USD ($) loan	Sep. 30, 2016 USD ($)	Jun. 30, 2017 USD ($)	Dec. 31, 2016 USD ($) loan	Jun. 30, 2016 USD ($)	Dec. 31, 2015 USD ($)
Investments, Debt and Equity Securities [Abstract]
Total residential whole loans		$ 1,700,000,000		$ 1,700,000,000			$ 1,400,000,000
Residential whole loans at carrying value	[1]	$ 639,216,000		$ 639,216,000			$ 590,540,000
Number of Residential whole loans at carrying value | loan		3,700		3,700			3,200
Allowance for loan losses, residential whole loan at carrying value		$ 318,000	$ 1,023,000	$ 318,000	$ 1,023,000	$ 375,000	$ 990,000	$ 1,023,000	$ 1,165,000
(Reversal) of provisions for loan losses		$ (57,000)	$ 0	$ (672,000)	$ (142,000)

[1] Includes approximately $131.3 million of Residential whole loans, at carrying value and $40.4 million of Residential whole loans, at fair value transferred to a consolidated VIE at September 30, 2017. Such assets can be used only to settle the obligations of the VIE.