Residential Mortgage Securities and MSR-Related Assets (Residential and Mortgage Securities) (Details) - USD ($)	Jun. 30, 2020	Dec. 31, 2019
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 166,670,000	$ 4,044,818,000
Purchase Premiums	2,052,000	67,108,000
Accretable Purchase Discounts	(8,559,000)	(90,694,000)
Discount Designated as Credit Reserve and OTTI	(21,437,000)	(436,598,000)
Amortized Cost	138,726,000	3,585,248,000
Gross Unrealized Gains	19,292,000	417,089,000
Gross Unrealized Losses	(9,675,000)	(18,818,000)
Net Unrealized Gain/(Loss)	9,617,000	398,271,000
Fair Value	148,343,000	3,983,519,000
Principal payments receivable		614,000
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		1,604,583,000
Purchase Premiums		62,790,000
Accretable Purchase Discounts		(22,000)
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		1,667,965,000
Gross Unrealized Gains		15,326,000
Gross Unrealized Losses		(18,709,000)
Net Unrealized Gain/(Loss)		(3,383,000)
Fair Value		1,664,582,000
Agency MBS | Fannie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		1,119,708,000
Purchase Premiums		43,249,000
Accretable Purchase Discounts		(22,000)
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		1,162,935,000
Gross Unrealized Gains		9,799,000
Gross Unrealized Losses		(14,741,000)
Net Unrealized Gain/(Loss)		(4,942,000)
Fair Value		1,157,993,000
Agency MBS | Freddie Mac
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		480,879,000
Purchase Premiums		19,468,000
Accretable Purchase Discounts		0
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		500,961,000
Gross Unrealized Gains		5,475,000
Gross Unrealized Losses		(3,968,000)
Net Unrealized Gain/(Loss)		1,507,000
Fair Value		502,468,000
Agency MBS | Ginnie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		3,996,000
Purchase Premiums		73,000
Accretable Purchase Discounts		0
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		4,069,000
Gross Unrealized Gains		52,000
Gross Unrealized Losses		0
Net Unrealized Gain/(Loss)		52,000
Fair Value		4,121,000
Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	62,361,000	2,195,303,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(8,430,000)	(90,617,000)
Discount Designated as Credit Reserve and OTTI	(669,000)	(436,598,000)
Amortized Cost	53,262,000	1,668,088,000
Gross Unrealized Gains	7,347,000	395,459,000
Gross Unrealized Losses	(6,372,000)	(18,000)
Net Unrealized Gain/(Loss)	975,000	395,441,000
Fair Value	54,237,000	2,063,529,000
Expected to Recover Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	59,512,000	722,477,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(8,430,000)	(16,661,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	51,082,000	705,816,000
Gross Unrealized Gains	6,962,000	19,861,000
Gross Unrealized Losses	(6,372,000)	(9,000)
Net Unrealized Gain/(Loss)	590,000	19,852,000
Fair Value	51,672,000	725,668,000
Expected to Recover Less Than Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,849,000	1,472,826,000
Purchase Premiums	0	0
Accretable Purchase Discounts	0	(73,956,000)
Discount Designated as Credit Reserve and OTTI	(669,000)	(436,598,000)
Amortized Cost	2,180,000	962,272,000
Gross Unrealized Gains	385,000	375,598,000
Gross Unrealized Losses	0	(9,000)
Net Unrealized Gain/(Loss)	385,000	375,589,000
Fair Value	$ 2,565,000	$ 1,337,861,000
Percentage of current face amount of Non-Agency MBS to be recovered	99.00%	80.00%
Total MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		$ 3,799,886,000
Purchase Premiums		62,790,000
Accretable Purchase Discounts		(90,639,000)
Discount Designated as Credit Reserve and OTTI		(436,598,000)
Amortized Cost		3,336,053,000
Gross Unrealized Gains		410,785,000
Gross Unrealized Losses		(18,727,000)
Net Unrealized Gain/(Loss)		392,058,000
Fair Value		3,728,111,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 104,309,000	244,932,000
Purchase Premiums	2,052,000	4,318,000
Accretable Purchase Discounts	(129,000)	(55,000)
Discount Designated as Credit Reserve and OTTI	(20,768,000)	0
Amortized Cost	85,464,000	249,195,000
Gross Unrealized Gains	11,945,000	6,304,000
Gross Unrealized Losses	(3,303,000)	(91,000)
Net Unrealized Gain/(Loss)	8,642,000	6,213,000
Fair Value	94,106,000	255,408,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	59,400,000	632,300,000
Amortized Cost	51,000,000.0	631,800,000
Fair Value	51,500,000	635,000,000.0
CRT, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	495,000	6,300,000
Gross Unrealized Losses	(3,300,000)	(91,000)
Fair Value	$ 63,400,000	255,400,000
Agency MBS, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains		4,500,000
Gross Unrealized Losses		0
Debt securities, fair value option		$ 280,300,000