Residential Mortgage Securities and MSR-Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Debt Securities, Available-for-sale [Line Items]
Interest income	$ 2,329	$ 60,639	$ 51,678	$ 211,676
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Debt Securities, Available-for-sale [Line Items]
Coupon interest	0	18,994	14,038	66,560
Effective yield adjustment	0	(7,188)	(5,186)	(21,039)
Interest income	0	11,806	8,852	45,521
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	42	21,011	18,222	68,144
Effective yield adjustment	6	10,336	10,564	38,003
Interest income	48	31,347	28,786	106,147
Accretion Income				3,100
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	811	13,227	7,622	44,305
Effective yield adjustment	94	8	449	158
Interest income	905	13,235	8,071	44,463
Accretion Income		4	277	152
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	956	5,174	6,063	16,769
Effective yield adjustment	420	(923)	(94)	(1,224)
Interest income	1,376	4,251	5,969	15,545
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	2,991	15,273	23,332	38,230
Effective yield adjustment	3,250	1	6,857	2
Interest income	$ 6,241	$ 15,274	$ 30,189	$ 38,232