Residential Mortgage Securities and MSR-Related Assets (Residential and Mortgage Securities) (Details) - USD ($)	9 Months Ended	12 Months Ended
	Sep. 30, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 4,641,832,000	$ 6,646,178,000
Purchase Premiums	74,178,000	112,369,000
Accretable Purchase Discounts	(102,437,000)	(154,942,000)
Discount Designated as Credit Reserve and OTTI	(462,095,000)	(516,116,000)
Amortized Cost	4,152,443,000	6,088,525,000
Gross Unrealized Gains	456,885,000	484,591,000
Gross Unrealized Losses	(19,774,000)	(63,783,000)
Net Unrealized Gain/(Loss)	437,111,000	420,808,000
Fair Value	4,589,554,000	6,509,333,000
Principal payments receivable	965,000	1,000,000.0
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,747,799,000	2,630,630,000
Purchase Premiums	68,296,000	103,008,000
Accretable Purchase Discounts	(23,000)	(24,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,817,037,000	2,734,650,000
Gross Unrealized Gains	16,108,000	13,061,000
Gross Unrealized Losses	(19,272,000)	(49,498,000)
Net Unrealized Gain/(Loss)	(3,164,000)	(36,437,000)
Fair Value	1,813,873,000	2,698,213,000
Agency MBS | Fannie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,217,508,000	1,716,340,000
Purchase Premiums	46,917,000	65,930,000
Accretable Purchase Discounts	(23,000)	(24,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,264,402,000	1,782,246,000
Gross Unrealized Gains	10,896,000	12,107,000
Gross Unrealized Losses	(15,282,000)	(32,321,000)
Net Unrealized Gain/(Loss)	(4,386,000)	(20,214,000)
Fair Value	1,260,016,000	1,762,032,000
Agency MBS | Freddie Mac
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	526,173,000	909,561,000
Purchase Premiums	21,303,000	36,991,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	548,441,000	947,588,000
Gross Unrealized Gains	5,159,000	907,000
Gross Unrealized Losses	(3,990,000)	(17,177,000)
Net Unrealized Gain/(Loss)	1,169,000	(16,270,000)
Fair Value	549,610,000	931,318,000
Agency MBS | Ginnie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	4,118,000	4,729,000
Purchase Premiums	76,000	87,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	4,194,000	4,816,000
Gross Unrealized Gains	53,000	47,000
Gross Unrealized Losses	0	0
Net Unrealized Gain/(Loss)	53,000	47,000
Fair Value	4,247,000	4,863,000
Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,529,749,000	3,538,804,000
Purchase Premiums	0	40,000
Accretable Purchase Discounts	(102,297,000)	(155,025,000)
Discount Designated as Credit Reserve and OTTI	(462,095,000)	(516,116,000)
Amortized Cost	1,965,357,000	2,867,703,000
Gross Unrealized Gains	432,693,000	458,985,000
Gross Unrealized Losses	(261,000)	(8,389,000)
Net Unrealized Gain/(Loss)	432,432,000	450,596,000
Fair Value	2,397,789,000	3,318,299,000
Credit reserve	451,100,000	503,300,000
OTTI	11,000,000.0	12,800,000
Expected to Recover Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	917,981,000	1,536,485,000
Purchase Premiums	0	40,000
Accretable Purchase Discounts	(14,176,000)	(21,725,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	903,805,000	1,514,800,000
Gross Unrealized Gains	17,905,000	20,520,000
Gross Unrealized Losses	(5,000)	(7,620,000)
Net Unrealized Gain/(Loss)	17,900,000	12,900,000
Fair Value	921,705,000	1,527,700,000
Expected to Recover Less Than Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,611,768,000	2,002,319,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(88,121,000)	(133,300,000)
Discount Designated as Credit Reserve and OTTI	(462,095,000)	(516,116,000)
Amortized Cost	1,061,552,000	1,352,903,000
Gross Unrealized Gains	414,788,000	438,465,000
Gross Unrealized Losses	(256,000)	(769,000)
Net Unrealized Gain/(Loss)	414,532,000	437,696,000
Fair Value	$ 1,476,084,000	$ 1,790,599,000
Percentage of current face amount of Non-Agency MBS to be recovered	82.00%	85.00%
Total MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 4,277,548,000	$ 6,169,434,000
Purchase Premiums	68,296,000	103,048,000
Accretable Purchase Discounts	(102,320,000)	(155,049,000)
Discount Designated as Credit Reserve and OTTI	(462,095,000)	(516,116,000)
Amortized Cost	3,782,394,000	5,602,353,000
Gross Unrealized Gains	448,801,000	472,046,000
Gross Unrealized Losses	(19,533,000)	(57,887,000)
Net Unrealized Gain/(Loss)	429,268,000	414,159,000
Fair Value	4,211,662,000	6,016,512,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	364,284,000	476,744,000
Purchase Premiums	5,882,000	9,321,000
Accretable Purchase Discounts	(117,000)	107,000
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	370,049,000	486,172,000
Gross Unrealized Gains	8,084,000	12,545,000
Gross Unrealized Losses	(241,000)	(5,896,000)
Net Unrealized Gain/(Loss)	7,843,000	6,649,000
Fair Value	377,892,000	492,821,000
Agency MBS, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	4,100,000	0
Gross Unrealized Losses	0	(3,300,000)
Debt Securities, Fair Value Option	312,500,000	736,500,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	824,800,000	1,400,000,000
Amortized Cost	824,300,000	1,400,000,000
Fair Value	828,100,000	1,400,000,000
CRT, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	7,800,000	12,500,000
Gross Unrealized Losses	(200,000)	(5,600,000)
Fair Value	$ 351,400,000	$ 477,400,000