Residential Whole Loans - Fair Value Components of Net Income (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Sep. 30, 2017	Sep. 30, 2016
Investments, Debt and Equity Securities [Abstract]
Coupon payments and other income received	$ 9,824	$ 6,253	$ 27,971	$ 15,987
Net unrealized gains	5,289	10,913	12,499	25,529
Net gain on payoff/liquidation of loans	1,456	1,535	3,076	3,536
Net gain on transfer to REO	2,110	938	5,114	2,677
Total	$ 18,679	$ 19,639	$ 48,660	$ 47,729