Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Receivables [Abstract]
Coupon payments, realized gains and other income received	$ 17,477	$ 22,202	$ 54,684	$ 67,966
Net unrealized gains/(losses)	58,863	13,185	(13,683)	33,312
Net gain on transfers to REO	531	4,788	3,430	15,637
Total	$ 76,871	$ 40,175	$ 44,431	$ 116,915