Residential Whole Loans - Provision for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 1,023	$ 587	$ 1,165	$ 137
Provisions/(reversal of provisions) for loan losses	0	335	(142)	785
Balance at the end of period	$ 1,023	$ 922	$ 1,023	$ 922