Residential Whole Loans - Fair Value (Details) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2017 USD ($) loan	Dec. 31, 2016 USD ($) loan
Investments, Debt and Equity Securities [Abstract]
Outstanding principal balance	$ 907,312	$ 966,276
Aggregate fair value	$ 775,152	$ 814,682
Number of loans | loan	3,558	3,812