Residential Mortgage Securities and MSR-Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Debt Securities, Available-for-sale [Line Items]
Interest income	$ 4,975	$ 72,395	$ 49,349	$ 151,037
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	402	22,938	14,038	47,566
Effective yield adjustment	(412)	(7,664)	(5,186)	(13,851)
Interest income	(10)	15,274	8,852	33,715
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	897	22,861	18,179	47,133
Effective yield adjustment	1,153	14,523	10,560	27,667
Interest income	2,050	37,384	28,739	74,800
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	1,228	14,635	6,811	31,078
Effective yield adjustment	77	8	355	150
Interest income	1,305	14,643	7,166	31,228
Accretion Income			277	148
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	1,622	5,477	5,107	11,595
Effective yield adjustment	8	(383)	(515)	(301)
Interest income	1,630	5,094	4,592	11,294
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	6,133	12,338	20,340	22,957
Effective yield adjustment	3,608	0	3,608	1
Interest income	$ 9,741	$ 12,338	$ 23,948	$ 22,958