Residential Whole Loans-Fair Value Components of Net Income (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Residential Whole Loans [Abstract]
Coupon payments and other income received									$ 41,373	$ 23,017	$ 9,304
Net unrealized gains									33,617	31,254	6,539
Net gain on payoff/liquidation of loans									4,958	5,413	1,879
Net gain on transfers to REO									10,071	2,921	1,853
Total	$ 41,359	$ 18,679	$ 16,208	$ 13,773	$ 14,876	$ 19,639	$ 15,742	$ 12,348	$ 90,019	$ 62,605	$ 19,575