Residential Whole Loans - Narrative (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Residential Whole Loans [Abstract]
Residential Whole Loans	$ 1,400,000				$ 895,100				$ 1,400,000	$ 895,100
Equity interest in certificates	100.00%								100.00%
Residential whole loans at carrying value	$ 590,540				271,845				$ 590,540	271,845
Allowance for loan leases	990				1,165				990	1,165	$ 137	$ 0
(Reversal of provisions)/provisions for loan losses									(175)	1,028	137
Net gain on residential whole loans held at fair value	$ 14,632	$ 18,701	$ 14,470	$ 11,881	$ 6,899	$ 5,565	$ 3,224	$ 2,034	$ 59,684	$ 17,722	$ 116