Residential Whole Loans - Provision for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 1,128	$ 464	$ 1,165	$ 137
(Reversal of)/provisions for loan losses	(105)	123	(142)	450
Balance at the end of period	$ 1,023	$ 587	$ 1,023	$ 587