Residential Whole Loans - Fair Value (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2016 USD ($) loan	Dec. 31, 2015 USD ($) loan
Investments, Debt and Equity Securities [Abstract]
Outstanding principal balance	$ 844,846	$ 786,330
Aggregate fair value	$ 684,582	$ 623,276
Number of loans | loan	3,117	3,143