Residential Mortgage Securities and MSR-Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	$ 22,938	$ 20,040	$ 47,566	$ 40,997
Effective yield adjustment	(7,664)	(6,870)	(13,851)	(12,534)
Interest income	15,274	13,170	33,715	28,463
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	22,861	27,931	47,133	56,765
Effective yield adjustment	14,523	17,462	27,667	34,664
Interest income	37,384	45,393	74,800	91,429
Accretion Income	3,100		3,100
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	14,635	9,588	31,078	19,641
Effective yield adjustment	8	62	150	75
Interest income	14,643	9,650	31,228	19,716
Accretion Income	148	40		1,200
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	5,477	7,854	11,595	16,227
Effective yield adjustment	(383)	841	(301)	1,964
Interest income	5,094	8,695	11,294	18,191
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	12,300	5,081	22,886	12,598
Effective yield adjustment	38	1,138	72	1,244
Interest income	$ 12,338	$ 6,219	$ 22,958	$ 13,842