Residential Mortgage Securities and MSR-Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019	Sep. 30, 2018
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	$ 18,994	$ 21,549	$ 66,560	$ 62,546
Effective yield adjustment	(7,188)	(7,217)	(21,039)	(19,751)
Interest income	11,806	14,332	45,521	42,795
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	21,011	27,026	68,144	83,791
Effective yield adjustment	10,336	18,984	38,003	53,648
Interest income	31,347	46,010	106,147	137,439
Accretion Income		2,300	3,100	2,300
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	13,227	11,526	44,305	31,167
Effective yield adjustment	8	1,131	158	1,206
Interest income	13,235	12,657	44,463	32,373
Accretion Income	4	1,100	152	1,200
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	5,174	7,257	16,769	23,484
Effective yield adjustment	(923)	491	(1,224)	2,455
Interest income	4,251	7,748	15,545	25,939
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	15,222	6,387	38,107	18,985
Effective yield adjustment	52	20	125	1,264
Interest income	$ 15,274	$ 6,407	$ 38,232	$ 20,249