Residential Whole Loans at Carrying Value-Provision for Loan Losses (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 990	$ 1,165	$ 137
(Reversal of provisions)/provisions for loan losses	(660)	(175)	1,028
Balance at the end of period	$ 330	$ 990	$ 1,165