Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2018	Sep. 30, 2017	Sep. 30, 2018	Sep. 30, 2017
Receivables [Abstract]
Coupon payments and other income received	$ 17,634	$ 9,824	$ 52,034	$ 27,971
Net unrealized gains	8,442	5,289	26,788	12,499
Net gain on payoff/liquidation of loans	3,251	1,456	10,203	3,076
Net gain on transfers to REO	5,615	2,110	16,858	5,114
Total	$ 34,942	$ 18,679	$ 105,883	$ 48,660