Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Receivables [Abstract]
Coupon payments and other income received									$ 82,168	$ 70,515	$ 41,399
Net unrealized gains									47,849	36,725	33,617
Net gain on payoff/liquidation of loans									9,270	11,087	4,958
Net gain on transfers to REO									19,043	19,292	10,071
Total	$ 41,415	$ 40,175	$ 51,473	$ 25,267	$ 31,736	$ 34,942	$ 32,443	$ 38,498	$ 158,330	$ 137,619	$ 90,045