Residential Whole Loans - Narrative (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015	Mar. 31, 2016	Dec. 31, 2015	Mar. 31, 2015	Dec. 31, 2014
Investments, Debt and Equity Securities [Abstract]
Residential whole loans	$ 1,100,000		$ 1,100,000			$ 895,100
Equity interest in certificates (percent)	100.00%		100.00%
Residential whole loans at carrying value	$ 392,172		$ 392,172			271,845
Allowance for loan losses, residential whole loan at carrying value	1,023	$ 587	1,023	$ 587	$ 1,128	$ 1,165	$ 464	$ 137
(Reversal of)/Provision for loan losses, residential whole loan at carrying value	$ (105)	$ 123	$ (142)	$ 450