Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Receivables [Abstract]
Coupon payments and other income received									$ 70,515	$ 41,399	$ 23,017
Net unrealized gains									36,725	33,617	31,254
Net gain on payoff/liquidation of loans									11,087	4,958	5,413
Net gain on transfers to REO									19,292	10,071	2,921
Total	$ 31,736	$ 34,942	$ 32,443	$ 38,498	$ 41,385	$ 18,679	$ 16,208	$ 13,773	$ 137,619	$ 90,045	$ 62,605