Residential Whole Loans - Carrying Value Accretable Yield Rollforward (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Residential Whole Loans Accretable Yield [Roll Forward]
Balance at beginning of period	$ 234,527	$ 155,574	$ 175,271	$ 133,012
Additions	28,801	3,620	98,550	34,901
Accretion	(5,917)	(3,972)	(16,112)	(11,406)
Reclassifications from/(to) non-accretable difference, net	218	2,692	(80)	1,407
Balance at end of period	257,629	$ 157,914	257,629	$ 157,914
Residential whole loans held at carrying value excluded, purchase transaction not occurred	$ 111,200		$ 111,200