Residential Whole Loans - Narrative (Details) $ in Thousands		3 Months Ended								12 Months Ended
		Dec. 31, 2017 USD ($) loan	Sep. 30, 2017 USD ($)	Jun. 30, 2017 USD ($)	Mar. 31, 2017 USD ($)	Dec. 31, 2016 USD ($) loan	Sep. 30, 2016 USD ($)	Jun. 30, 2016 USD ($)	Mar. 31, 2016 USD ($)	Dec. 31, 2017 USD ($) loan	Dec. 31, 2016 USD ($) loan	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)
Residential Whole Loans [Abstract]
Residential whole loans		$ 2,200,000				$ 1,400,000				$ 2,200,000	$ 1,400,000
Residential whole loans at carrying value	[1]	$ 908,516				$ 590,540				$ 908,516	$ 590,540
Residential whole loans | loan		4,800				3,200				4,800	3,200
Allowance for loan leases		$ 330				$ 990				$ 330	$ 990	$ 1,165	$ 137
(Reversal of provisions)/provisions for loan losses										(660)	(175)	1,028
Net gain on residential whole loans held at fair value		$ 41,359	$ 18,679	$ 16,208	$ 13,773	$ 14,876	$ 19,639	$ 15,742	$ 12,348	$ 90,019	$ 62,605	$ 19,575

[1] Includes approximately $183.2 million of Residential whole loans, at carrying value and $289.3 million of Residential whole loans, at fair value transferred to consolidated VIEs at December 31, 2017. Such assets can be used only to settle the obligations of each respective VIE.