Collateral Positions (Details 2) $ in Thousands	Dec. 31, 2016 USD ($)
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	$ 10,643,058
Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	10,534,089
Amortized Cost	10,002,330
Accrued Interest on Pledged Assets	22,585
Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	86,317
Amortized Cost	87,065
Accrued Interest on Pledged Assets	67
Agency MBS
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	3,549,122
Agency MBS | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	3,507,933
Amortized Cost	3,488,904
Accrued Interest on Pledged Assets	8,654
Agency MBS | Assets Pledged Under Repurchase Agreements and Other Advances | Federal Home Loan Bank Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	227,200
Amortized Cost	226,600
Accrued Interest on Pledged Assets	597
Agency MBS | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	32,468
Amortized Cost	33,216
Accrued Interest on Pledged Assets	67
Legacy Non-Agency MBS
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	2,326,321
Legacy Non-Agency MBS | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	2,317,708
Amortized Cost	1,841,401
Accrued Interest on Pledged Assets	8,613
Non-Agency MBS pledged as collateral in connection with contemporaneous repurchase and reverse repurchase agreements entered into with a single counterparty	688,200
Legacy Non-Agency MBS | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
3 Year Step-up securities
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	2,662,339
3 Year Step-up securities | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	2,660,491
Amortized Cost	2,657,726
Accrued Interest on Pledged Assets	1,848
3 Year Step-up securities | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
U.S. Treasury securities
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	510,767
U.S. Treasury securities | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	510,767
Amortized Cost	510,767
Accrued Interest on Pledged Assets	0
U.S. Treasury securities | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
CRT securities
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	357,710
CRT securities | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	357,488
Amortized Cost	336,706
Accrued Interest on Pledged Assets	222
CRT securities | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Residential whole loans
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	1,178,336
Residential whole loans | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	1,175,088
Amortized Cost	1,162,212
Accrued Interest on Pledged Assets	3,248
Residential whole loans | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Cash
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	58,463
Cash | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	4,614
Amortized Cost	4,614
Accrued Interest on Pledged Assets	0
Cash | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	53,849
Amortized Cost	53,849
Accrued Interest on Pledged Assets	0
Fair Value, Estimate Not Practicable, Carrying (Reported) Amount | Residential whole loans | Assets Pledged Under Repurchase Agreements and Other Advances
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	440,800
Amortized Cost	427,900
Total Fair Value of Assets Pledged and Accrued Interest	$ 732,400