MBS, CRT Securities and MSR Related Assets (Interest Income) (Details) - USD ($)	3 Months Ended		9 Months Ended
	Sep. 30, 2018	Sep. 30, 2017	Sep. 30, 2018	Sep. 30, 2017
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	$ 21,549,000	$ 23,473,000	$ 62,546,000	$ 74,589,000
Effective yield adjustment	(7,217,000)	(7,940,000)	(19,751,000)	(24,575,000)
Interest income	14,332,000	15,533,000	42,795,000	50,014,000
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	27,026,000	30,688,000	83,791,000	97,796,000
Effective yield adjustment	18,984,000	18,005,000	53,648,000	59,033,000
Interest income	46,010,000	48,693,000	137,439,000	156,829,000
Accretion Income	2,300,000		2,300,000	1,700,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Coupon interest	11,526,000	13,947,000	31,167,000	54,475,000
Effective yield adjustment	1,131,000	612,000	1,206,000	1,424,000
Interest income	12,657,000	14,559,000	32,373,000	55,899,000
Accretion Income	1,100,000	575,000	1,200,000	1,200,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Coupon interest	7,257,000	7,868,000	23,484,000	19,712,000
Effective yield adjustment	491,000	808,000	2,455,000	3,186,000
Interest income	7,748,000	8,676,000	25,939,000	22,898,000
MSR related assets
Debt Securities, Available-for-sale [Line Items]
Coupon interest	6,407,000	7,117,000	19,005,000	17,621,000
Effective yield adjustment	0	77,000	1,244,000	212,000
Interest income	$ 6,407,000	$ 7,194,000	$ 20,249,000	$ 17,833,000