Other Assets - Derivative Instruments (Details 2) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017 USD ($)	Sep. 30, 2017 USD ($)	Jun. 30, 2017 USD ($)	Mar. 31, 2017 USD ($)	Dec. 31, 2016 USD ($)	Sep. 30, 2016 USD ($)	Jun. 30, 2016 USD ($)	Mar. 31, 2016 USD ($)	Dec. 31, 2017 USD ($) derivative	Dec. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)	Dec. 31, 2017 USD ($)	Dec. 31, 2016 USD ($)
Derivative [Line Items]
Assets Pledged												$ 8,145,197	$ 10,620,406
Number of terminated derivatives | derivative									0
Net unrealized gains (losses)	$ (11,424)			$ (46,721)	$ (46,721)			$ (69,399)	$ (46,721)	$ (69,399)	$ (59,062)	(11,424)	(46,721)
Interest expense attributable to Swaps	48,495	$ 49,275	$ 49,022	50,349	49,868	$ 48,167	$ 47,720	47,600	197,141	193,355	176,948
AOCI from derivative hedging instruments:
Balance at beginning of period				$ (46,721)				$ (69,399)	(46,721)	(69,399)	(59,062)
Net gain/(loss) on Swaps									35,297	22,678	(10,337)
Balance at end of period	$ (11,424)				$ (46,721)				$ (11,424)	(46,721)	(69,399)
Interest Rate Contract
Derivative [Line Items]
Assets Pledged												28,161	86,317
Interest Rate Contract | Agency MBS, at fair value
Derivative [Line Items]
Assets Pledged												21,756	32,468
Interest Rate Contract | Restricted cash
Derivative [Line Items]
Assets Pledged												6,405	53,849
Swaps, at fair value
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 2,550,000	$ 2,900,000
Average maturity term of swaps									27 months
Weighted average fixed-pay rate												2.04%	1.87%
Weighted Average Variable Interest Rate												1.50%	0.72%
Interest expense attributable to Swaps									$ 24,524	$ 40,898	$ 53,759
Weighted average Swap rate paid									1.98%	1.82%	1.86%
Weighted average Swap rate received									1.07%	0.48%	0.19%
Swaps, at fair value | Within 30 days
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 0
Weighted average fixed-pay rate												0.00%	0.00%
Weighted Average Variable Interest Rate												0.00%	0.00%
Swaps, at fair value | Over 30 days to 3 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 50,000
Weighted average fixed-pay rate												0.00%	0.67%
Weighted Average Variable Interest Rate												0.00%	0.64%
Swaps, at fair value | Over 3 months to 6 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 50,000	$ 300,000
Weighted average fixed-pay rate												1.45%	0.57%
Weighted Average Variable Interest Rate												1.56%	0.66%
Swaps, at fair value | Over 6 months to 12 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 500,000	$ 0
Weighted average fixed-pay rate												1.50%	0.00%
Weighted Average Variable Interest Rate												1.46%	0.00%
Swaps, at fair value | Over 12 months to 24 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 200,000	$ 550,000
Weighted average fixed-pay rate												1.71%	1.49%
Weighted Average Variable Interest Rate												1.54%	0.71%
Swaps, at fair value | Over 24 months to 36 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 1,500,000	$ 200,000
Weighted average fixed-pay rate												2.22%	1.71%
Weighted Average Variable Interest Rate												1.51%	0.76%
Swaps, at fair value | Over 36 months to 48 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 200,000	$ 1,500,000
Weighted average fixed-pay rate												2.20%	2.22%
Weighted Average Variable Interest Rate												1.53%	0.74%
Swaps, at fair value | Over 48 months to 60 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 200,000
Weighted average fixed-pay rate												0.00%	2.20%
Weighted Average Variable Interest Rate												0.00%	0.75%
Swaps, at fair value | Over 60 months to 72 months (3)
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 100,000	$ 0
Weighted average fixed-pay rate												2.75%	0.00%
Weighted Average Variable Interest Rate												1.50%	0.00%
Swaps, at fair value | Over 72 months to 84 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 100,000
Weighted average fixed-pay rate												0.00%	2.75%
Weighted Average Variable Interest Rate												0.00%	0.74%
Swaps, at fair value | Hedging
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 2,600,000
Swaps, at fair value | Minimum | LIBOR
Derivative [Line Items]
Derivative, variable interest rate, term									1 month
Swaps, at fair value | Maximum
Derivative [Line Items]
Maximum maturity term of swaps									68 months
Swaps, at fair value | Maximum | LIBOR
Derivative [Line Items]
Derivative, variable interest rate, term									3 months
New Swaps:
Derivative [Line Items]
Aggregate notional amount									$ 0
Weighted average fixed-pay rate												0.00%
Number of new Swaps | derivative									0
Swaps amortized/expired:
Derivative [Line Items]
Aggregate notional amount									$ 350,000
Swaps amortized/expired: | YTD
Derivative [Line Items]
Weighted average fixed-pay rate												0.58%