Collateral Positions (Details 2) $ in Thousands	Jun. 30, 2017 USD ($)
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	$ 8,654,643
Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	3,133,830
Legacy Non-Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,987,955
3 Year Step-up securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,569,088
U.S. Treasury securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	370,837
CRT securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	517,390
Residential whole loans
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	962,863
Other investments
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	100,837
Cash
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	11,843
Repurchase Agreements and Other Advances
Collateral Positions
Fair Value	8,605,192
Amortized Cost	8,110,819
Accrued Interest on Pledged Assets	19,930
Repurchase Agreements and Other Advances | Agency MBS
Collateral Positions
Fair Value	3,101,432
Amortized Cost	3,100,753
Accrued Interest on Pledged Assets	7,916
Repurchase Agreements and Other Advances | Legacy Non-Agency MBS
Collateral Positions
Fair Value	1,980,719
Amortized Cost	1,550,498
Accrued Interest on Pledged Assets	7,236
Non-Agency MBS pledged as collateral in connection with contemporaneous repurchase and reverse repurchase agreements entered into with a single counterparty	692,000
Repurchase Agreements and Other Advances | 3 Year Step-up securities
Collateral Positions
Fair Value	1,567,914
Amortized Cost	1,563,770
Accrued Interest on Pledged Assets	1,174
Repurchase Agreements and Other Advances | U.S. Treasury securities
Collateral Positions
Fair Value	370,837
Amortized Cost	373,713
Accrued Interest on Pledged Assets	0
Repurchase Agreements and Other Advances | CRT securities
Collateral Positions
Fair Value	517,067
Amortized Cost	473,421
Accrued Interest on Pledged Assets	323
Repurchase Agreements and Other Advances | Residential whole loans
Collateral Positions
Fair Value	960,218
Amortized Cost	941,659
Accrued Interest on Pledged Assets	2,645
Repurchase Agreements and Other Advances | Other investments
Collateral Positions
Fair Value	100,201
Amortized Cost	100,201
Accrued Interest on Pledged Assets	636
Repurchase Agreements and Other Advances | Cash
Collateral Positions
Fair Value	6,804
Amortized Cost	6,804
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments
Collateral Positions
Fair Value	29,471
Amortized Cost	30,135
Accrued Interest on Pledged Assets	50
Derivative Hedging Instruments | Agency MBS
Collateral Positions
Fair Value	24,432
Amortized Cost	25,096
Accrued Interest on Pledged Assets	50
Derivative Hedging Instruments | Legacy Non-Agency MBS
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | 3 Year Step-up securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | U.S. Treasury securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | CRT securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Residential whole loans
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Other investments
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Cash
Collateral Positions
Fair Value	5,039
Amortized Cost	5,039
Accrued Interest on Pledged Assets	0
Carrying Value | Repurchase Agreements and Other Advances | Residential whole loans
Collateral Positions
Fair Value	289,100
Amortized Cost	270,600
Total Fair Value of Assets Pledged and Accrued Interest	$ 671,100