Residential Mortgage Securities and MSR-Related Assets (Residential and Mortgage Securities) (Details) - USD ($)	Sep. 30, 2020	Dec. 31, 2019
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 164,458,000	$ 4,044,818,000
Purchase Premiums	2,414,000	67,108,000
Accretable Purchase Discounts	(8,315,000)	(90,694,000)
Discount Designated as Credit Reserve and OTTI	(21,437,000)	(436,598,000)
Amortized Cost	137,120,000	3,585,248,000
Gross Unrealized Gains	21,737,000	417,089,000
Gross Unrealized Losses	(6,092,000)	(18,818,000)
Net Unrealized Gain/(Loss)	15,645,000	398,271,000
Fair  Value	152,765,000	3,983,519,000
Principal payments receivable		614,000
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		1,604,583,000
Purchase Premiums		62,790,000
Accretable Purchase Discounts		(22,000)
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		1,667,965,000
Gross Unrealized Gains		15,326,000
Gross Unrealized Losses		(18,709,000)
Net Unrealized Gain/(Loss)		(3,383,000)
Fair  Value		1,664,582,000
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member] | Fannie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		1,119,708,000
Purchase Premiums		43,249,000
Accretable Purchase Discounts		(22,000)
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		1,162,935,000
Gross Unrealized Gains		9,799,000
Gross Unrealized Losses		(14,741,000)
Net Unrealized Gain/(Loss)		(4,942,000)
Fair  Value		1,157,993,000
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member] | Freddie Mac
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		480,879,000
Purchase Premiums		19,468,000
Accretable Purchase Discounts		0
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		500,961,000
Gross Unrealized Gains		5,475,000
Gross Unrealized Losses		(3,968,000)
Net Unrealized Gain/(Loss)		1,507,000
Fair  Value		502,468,000
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member] | Ginnie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		3,996,000
Purchase Premiums		73,000
Accretable Purchase Discounts		0
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		4,069,000
Gross Unrealized Gains		52,000
Gross Unrealized Losses		0
Net Unrealized Gain/(Loss)		52,000
Fair  Value		4,121,000
Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	60,295,000	2,195,303,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(8,246,000)	(90,617,000)
Discount Designated as Credit Reserve and OTTI	(669,000)	(436,598,000)
Amortized Cost	51,380,000	1,668,088,000
Gross Unrealized Gains	8,015,000	395,459,000
Gross Unrealized Losses	(2,965,000)	(18,000)
Net Unrealized Gain/(Loss)	5,050,000	395,441,000
Fair  Value	$ 56,430,000	2,063,529,000
Expected to Recover Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		722,477,000
Purchase Premiums		0
Accretable Purchase Discounts		(16,661,000)
Discount Designated as Credit Reserve and OTTI		0
Amortized Cost		705,816,000
Gross Unrealized Gains		19,861,000
Gross Unrealized Losses		(9,000)
Net Unrealized Gain/(Loss)		19,852,000
Fair  Value		725,668,000
Expected to Recover Less Than Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		1,472,826,000
Purchase Premiums		0
Accretable Purchase Discounts		(73,956,000)
Discount Designated as Credit Reserve and OTTI		(436,598,000)
Amortized Cost		962,272,000
Gross Unrealized Gains		375,598,000
Gross Unrealized Losses		(9,000)
Net Unrealized Gain/(Loss)		375,589,000
Fair  Value		$ 1,337,861,000
Percentage of current face amount of Non-Agency MBS to be recovered	99.00%	80.00%
Total MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face		$ 3,799,886,000
Purchase Premiums		62,790,000
Accretable Purchase Discounts		(90,639,000)
Discount Designated as Credit Reserve and OTTI		(436,598,000)
Amortized Cost		3,336,053,000
Gross Unrealized Gains		410,785,000
Gross Unrealized Losses		(18,727,000)
Net Unrealized Gain/(Loss)		392,058,000
Fair  Value		3,728,111,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 104,163,000	244,932,000
Purchase Premiums	2,414,000	4,318,000
Accretable Purchase Discounts	(69,000)	(55,000)
Discount Designated as Credit Reserve and OTTI	(20,768,000)	0
Amortized Cost	85,740,000	249,195,000
Gross Unrealized Gains	13,722,000	6,304,000
Gross Unrealized Losses	(3,127,000)	(91,000)
Net Unrealized Gain/(Loss)	10,595,000	6,213,000
Fair  Value	96,335,000	255,408,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	57,400,000	632,300,000
Amortized Cost	49,200,000	631,800,000
Fair  Value	53,800,000	635,000,000.0
CRT, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	410,000	6,300,000
Gross Unrealized Losses	(3,100,000)	(91,000)
Fair  Value	$ 63,300,000	255,400,000
Agency MBS, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains		4,500,000
Gross Unrealized Losses		0
Debt securities, fair value option		$ 280,300,000