Investments in Debt Securities Investments in Debt Securities (Details)	3 Months Ended			9 Months Ended
	Sep. 30, 2020 USD ($)		Sep. 30, 2019 USD ($)	Sep. 30, 2020 USD ($)		Sep. 30, 2019 USD ($)	Dec. 31, 2019 USD ($)
Investments [Line Items]
Par value of available for sale debt security	$ 2,433,207,000			$ 2,433,207,000			$ 4,455,808,000
Investments, Unamortized Premium (Discount)	473,728,000			473,728,000			558,549,000
Debt Securities, Available-for-sale, Amortized Cost	2,906,935,000			2,906,935,000			5,014,357,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Gain, before Tax	90,253,000			90,253,000			175,157,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Loss, before Tax	(1,528,000)			(1,528,000)			(1,351,000)
Debt Securities, Available-for-sale	2,995,660,000			2,995,660,000			5,188,163,000
Proceeds from Sale of Available-for-sale Securities	403,095,000		$ 600,514,000	4,403,054,000		$ 1,033,066,000
Debt Securities, Available-for-sale, Realized Gain (Loss)	20,846,000		4,605,000	298,728,000		(5,755,000)
Debt Securities, Available-for-sale, Allowance for Credit Loss	0			0
Agency MBS
Investments [Line Items]
Notional balance for interest only securities	12,344,492,000			12,344,492,000			13,404,824,000
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	36,664,000			36,664,000			215,792,000
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than 12 Months, Accumulated Loss	$ 503,000			$ 503,000			$ 1,139,000
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions, Less than One Year	16			16			27
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	$ 689,000			$ 689,000			$ 75,745,000
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Accumulated Loss	$ 107,000			$ 107,000			$ 35,000
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions, Greater than or Equal to One Year	3			3			2
Interest-Only-Strip
Investments [Line Items]
Par value of available for sale debt security	$ 0	[1]		$ 0	[1]		$ 0	[2]
Investments, Unamortized Premium (Discount)	405,736,000			405,736,000			488,145,000
Debt Securities, Available-for-sale, Amortized Cost	405,736,000			405,736,000			488,145,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Gain, before Tax	9,497,000			9,497,000			11,760,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Loss, before Tax	(1,480,000)			(1,480,000)			(863,000)
Debt Securities, Available-for-sale	413,753,000			413,753,000			499,042,000
Non-Agency MBS
Investments [Line Items]
Par value of available for sale debt security	1,630,000			1,630,000			1,938,000
Investments, Unamortized Premium (Discount)	(517,000)			(517,000)			(780,000)
Notional balance for interest only securities	9,485,281,000			9,485,281,000			9,799,629,000
Debt Securities, Available-for-sale, Amortized Cost	1,113,000			1,113,000			1,158,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Gain, before Tax	333,000			333,000			552,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Loss, before Tax	(48,000)			(48,000)			(20,000)
Debt Securities, Available-for-sale	1,398,000			1,398,000			1,690,000
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	46,835,000			46,835,000			13,607,000
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than 12 Months, Accumulated Loss	$ 897,000			$ 897,000			$ 146,000
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions, Less than One Year	30			30			7
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	$ 100,000			$ 100,000			$ 1,099,000
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Accumulated Loss	$ 21,000			$ 21,000			$ 31,000
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions, Greater than or Equal to One Year	4			4			5
Agency | Commercial Mortgage Backed Securities
Investments [Line Items]
Par value of available for sale debt security	$ 271,764,000			$ 271,764,000			$ 1,890,186,000
Investments, Unamortized Premium (Discount)	3,445,000			3,445,000			15,414,000
Debt Securities, Available-for-sale, Amortized Cost	275,209,000			275,209,000			1,905,600,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Gain, before Tax	23,495,000			23,495,000			93,763,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Loss, before Tax	0			0			(6,000)
Debt Securities, Available-for-sale	298,704,000			298,704,000			1,999,357,000
Proceeds from Sale of Available-for-sale Securities	403,095,000		0	2,215,098,000		213,199,000
Debt Securities, Available-for-sale, Realized Loss	20,845,000		0	222,904,000		(6,493,000)
Agency | Residential Mortgage Backed Securities
Investments [Line Items]
Par value of available for sale debt security	2,159,813,000			2,159,813,000			2,563,684,000
Investments, Unamortized Premium (Discount)	65,064,000			65,064,000			55,770,000
Debt Securities, Available-for-sale, Amortized Cost	2,224,877,000			2,224,877,000			2,619,454,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Gain, before Tax	56,928,000			56,928,000			69,082,000
Debt Securities, Available-for-sale, Accumulated Gross Unrealized Loss, before Tax	0			0			(462,000)
Debt Securities, Available-for-sale	2,281,805,000			2,281,805,000			$ 2,688,074,000
Proceeds from Sale of Available-for-sale Securities	0		591,206,000	2,187,956,000		796,699,000
Debt Securities, Available-for-sale, Realized Gain	$ 1,000	[3]		$ 75,824,000
Debt Securities, Available-for-sale, Realized Loss			4,458,000			506,000
Agency | Interest-Only-Strip
Investments [Line Items]
Proceeds from Sale of Available-for-sale Securities			9,308,000			23,168,000
Debt Securities, Available-for-sale, Realized Loss			$ 147,000			$ 232,000

[1] The notional balance for Agency CMBS IO and non-Agency CMBS IO was $12,344,492 and $9,485,281 respectively, as of September 30, 2020.

[2] The notional balance for the Agency CMBS IO and non-Agency CMBS IO was $13,404,824 and $9,799,629, respectively, as of December 31, 2019.

[3] Additional realized gain for Agency RMBS relates to change in effective rate due to paydown that occurred between trade date of sale in second quarter of 2020 and settlement date in third quarter.