Residential Mortgage Securities and MSR Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	$ 82,446	$ 88,233	$ 96,678
Effective yield adjustment	(26,545)	(25,930)	(31,323)
Interest income	55,901	62,303	65,355
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	87,024	109,714	127,645
Effective yield adjustment	59,622	69,309	76,005
Interest income	146,646	179,023	203,650
Accretion income	14,500	2,700	1,700
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	53,086	46,339	65,957
Effective yield adjustment	338	1,434	1,505
Interest income	53,424	47,773	67,462
Accretion income	329	1,400	1,200
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	20,532	30,628	27,706
Effective yield adjustment	(1,949)	2,748	4,009
Interest income	18,583	33,376	31,715
MSR related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	52,644	27,174	24,534
Effective yield adjustment	3	1,246	296
Interest income	$ 52,647	$ 28,420	$ 24,830