Residential Morgage Securities and MSR Related Assets (MBS and CRT Securities) (Details) - USD ($)	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 6,646,178,000	$ 7,060,004,000
Purchase Premiums	112,369,000	113,003,000
Accretable Purchase Discounts	(154,942,000)	(218,915,000)
Discount Designated as Credit Reserve and OTTI	(516,116,000)	(593,227,000)
Amortized Cost	6,088,525,000	6,362,756,000
Fair Value	6,509,333,000	7,023,050,000
Gross Unrealized Gains	484,591,000	703,907,000
Gross Unrealized Losses	63,783,000	43,613,000
Net Unrealized Gain/(Loss)	420,808,000	660,294,000
Principal payments receivable	1,000,000	1,900,000
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,630,630,000	2,738,462,000
Purchase Premiums	103,008,000	104,066,000
Accretable Purchase Discounts	(24,000)	(40,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	2,734,650,000	2,844,379,000
Fair Value	2,698,213,000	2,824,681,000
Gross Unrealized Gains	13,061,000	23,439,000
Gross Unrealized Losses	49,498,000	43,137,000
Net Unrealized Gain/(Loss)	(36,437,000)	(19,698,000)
Agency MBS | Fannie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,716,340,000	2,170,974,000
Purchase Premiums	65,930,000	82,271,000
Accretable Purchase Discounts	(24,000)	(40,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,782,246,000	2,253,205,000
Fair Value	1,762,032,000	2,246,600,000
Gross Unrealized Gains	12,107,000	21,736,000
Gross Unrealized Losses	32,321,000	28,341,000
Net Unrealized Gain/(Loss)	(20,214,000)	(6,605,000)
Agency MBS | Freddie Mac
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	909,561,000	561,346,000
Purchase Premiums	36,991,000	21,683,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	947,588,000	584,920,000
Fair Value	931,318,000	571,748,000
Gross Unrealized Gains	907,000	1,624,000
Gross Unrealized Losses	17,177,000	14,796,000
Net Unrealized Gain/(Loss)	(16,270,000)	(13,172,000)
Agency MBS | Ginnie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	4,729,000	6,142,000
Purchase Premiums	87,000	112,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	4,816,000	6,254,000
Fair Value	4,863,000	6,333,000
Gross Unrealized Gains	47,000	79,000
Gross Unrealized Losses	0	0
Net Unrealized Gain/(Loss)	47,000	79,000
Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	3,538,804,000	3,718,743,000
Purchase Premiums	40,000	50,000
Accretable Purchase Discounts	(155,025,000)	(215,325,000)
Discount Designated as Credit Reserve and OTTI	(516,116,000)	(593,227,000)
Amortized Cost	2,867,703,000	2,910,241,000
Fair Value	3,318,299,000	3,533,966,000
Gross Unrealized Gains	458,985,000	624,178,000
Gross Unrealized Losses	8,389,000	453,000
Net Unrealized Gain/(Loss)	450,596,000	623,725,000
Credit reserve	503,300,000	579,000,000
OTTI	12,800,000	14,200,000
Non-Agency MBS | Expected to Recover Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,536,485,000	1,128,808,000
Purchase Premiums	40,000	50,000
Accretable Purchase Discounts	(21,725,000)	(22,737,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,514,800,000	1,106,121,000
Fair Value	1,527,700,000	1,132,205,000
Gross Unrealized Gains	20,520,000	26,518,000
Gross Unrealized Losses	7,620,000	434,000
Net Unrealized Gain/(Loss)	12,900,000	26,084,000
Non-Agency MBS | Expected to Recover Less Than Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,002,319,000	2,589,935,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(133,300,000)	(192,588,000)
Discount Designated as Credit Reserve and OTTI	(516,116,000)	(593,227,000)
Amortized Cost	1,352,903,000	1,804,120,000
Fair Value	1,790,599,000	2,401,761,000
Gross Unrealized Gains	438,465,000	597,660,000
Gross Unrealized Losses	769,000	19,000
Net Unrealized Gain/(Loss)	$ 437,696,000	$ 597,641,000
Percentage of current face amount of Non-Agency MBS to be recovered	85.00%	84.00%
Total MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 6,169,434,000	$ 6,457,205,000
Purchase Premiums	103,048,000	104,116,000
Accretable Purchase Discounts	(155,049,000)	(215,365,000)
Discount Designated as Credit Reserve and OTTI	(516,116,000)	(593,227,000)
Amortized Cost	5,602,353,000	5,754,620,000
Fair Value	6,016,512,000	6,358,647,000
Gross Unrealized Gains	472,046,000	647,617,000
Gross Unrealized Losses	57,887,000	43,590,000
Net Unrealized Gain/(Loss)	414,159,000	604,027,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	476,744,000	602,799,000
Purchase Premiums	9,321,000	8,887,000
Accretable Purchase Discounts	107,000	(3,550,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	486,172,000	608,136,000
Fair Value	492,821,000	664,403,000
Gross Unrealized Gains	12,545,000	56,290,000
Gross Unrealized Losses	5,896,000	23,000
Net Unrealized Gain/(Loss)	6,649,000	56,267,000
Agency MBS, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Fair Value	736,500,000
Gross Unrealized Gains	0
Gross Unrealized Losses	3,300,000
Debt securities, fair value option	736,500,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,400,000,000	922,000,000
Amortized Cost	1,400,000,000	920,100,000
Fair Value	1,400,000,000	923,100,000
CRT, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Fair Value	477,400,000	528,900,000
Gross Unrealized Gains	12,500,000	40,500,000
Gross Unrealized Losses	$ 5,600,000	$ 23,000