Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Receivables [Abstract]
Coupon payments, realized gains and other income received	$ 16,676	$ 19,036
Net unrealized gains/(losses)	32,088	(74,556)
Net gain on transfers to REO	1,045	2,760
Total	$ 49,809	$ (52,760)