Fair Value of Financial Instruments - Assets and Liabilities on Recurring Basis (Details) - USD ($)	Dec. 31, 2016	Dec. 31, 2015
Assets:
MBS and CRT securities	$ 9,969,163,000	$ 11,356,643,000
Securities obtained and pledged as collateral, at fair value	510,767,000	507,443,000
Residential whole loans, at fair value	814,682,000	623,300,000
Swaps	233,000	1,127,000
Liabilities:
Obligation to return securities obtained as collateral	510,767,000	507,443,000
Agency MBS
Assets:
MBS and CRT securities	3,738,497,000	4,752,244,000
Non-Agency MBS
Assets:
MBS and CRT securities	5,825,816,000	6,420,817,000
CRT securities
Assets:
MBS and CRT securities	404,850,000	183,582,000
Level 2
Components of financial instruments carried at fair value
Derivative credit risk valuation adjustment	0
Recurring basis
Assets:
Securities obtained and pledged as collateral, at fair value	510,767,000	507,443,000
Swaps	233,000	1,127,000
Total assets carried at fair value	11,294,845,000	12,488,489,000
Liabilities:
Swaps	46,954,000	70,526,000
Obligation to return securities obtained as collateral	510,767,000	507,443,000
Total liabilities carried at fair value	557,721,000	577,969,000
Recurring basis \| Agency MBS
Assets:
MBS and CRT securities	3,738,497,000	4,752,244,000
Recurring basis \| Non-Agency MBS
Assets:
MBS and CRT securities	5,825,816,000	6,420,817,000
Recurring basis \| CRT securities
Assets:
MBS and CRT securities	404,850,000	183,582,000
Recurring basis \| Residential whole loans
Assets:
Residential whole loans, at fair value	814,682,000	623,276,000
Recurring basis \| Level 1
Assets:
Securities obtained and pledged as collateral, at fair value	510,767,000	507,443,000
Swaps	0	0
Total assets carried at fair value	510,767,000	507,443,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	510,767,000	507,443,000
Total liabilities carried at fair value	510,767,000	507,443,000
Recurring basis \| Level 1 \| Agency MBS
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 1 \| Non-Agency MBS
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 1 \| CRT securities
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 1 \| Residential whole loans
Assets:
Residential whole loans, at fair value	0	0
Recurring basis \| Level 2
Assets:
Securities obtained and pledged as collateral, at fair value	0	0
Swaps	233,000	1,127,000
Total assets carried at fair value	9,969,396,000	11,357,770,000
Liabilities:
Swaps	46,954,000	70,526,000
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	46,954,000	70,526,000
Recurring basis \| Level 2 \| Agency MBS
Assets:
MBS and CRT securities	3,738,497,000	4,752,244,000
Recurring basis \| Level 2 \| Non-Agency MBS
Assets:
MBS and CRT securities	5,825,816,000	6,420,817,000
Recurring basis \| Level 2 \| CRT securities
Assets:
MBS and CRT securities	404,850,000	183,582,000
Recurring basis \| Level 2 \| Residential whole loans
Assets:
Residential whole loans, at fair value	0	0
Recurring basis \| Level 3
Assets:
Securities obtained and pledged as collateral, at fair value	0	0
Swaps	0	0
Total assets carried at fair value	814,682,000	623,276,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	0
Recurring basis \| Level 3 \| Agency MBS
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 3 \| Non-Agency MBS
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 3 \| CRT securities
Assets:
MBS and CRT securities	0	0
Recurring basis \| Level 3 \| Residential whole loans
Assets:
Residential whole loans, at fair value	$ 814,682,000	$ 623,276,000

Fair value portion of probable future economic benefits obtained or controlled by an entity as a result of past transactions or events.

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 820
-SubTopic 10
-Section 50
-Paragraph 2
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=66048111&loc=d3e19207-110258

+ Details

Name:	us-gaap_AssetsFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- References

No definition available.

+ Details

Name:	us-gaap_AssetsFairValueDisclosureAbstract
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Amount of investment in debt and equity securities categorized neither as held-to-maturity nor trading.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 320
-SubTopic 10
-Section 25
-Paragraph 1
-Subparagraph (b)
-URI http://asc.fasb.org/extlink&oid=65884222&loc=d3e22054-111558

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 320
-SubTopic 10
-Section 50
-Paragraph 5
-Subparagraph (aa)
-URI http://asc.fasb.org/extlink&oid=73527792&loc=d3e27232-111563

Reference 3: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 320
-SubTopic 10
-Section 45
-Paragraph 1
-URI http://asc.fasb.org/extlink&oid=66010992&loc=d3e26610-111562

+ Details

Name:	us-gaap_AvailableForSaleSecurities
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value at acquisition of acquired loans not accounted for as debt securities with evidence of deterioration of credit quality.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 30
-Section 15
-Paragraph 2
-URI http://asc.fasb.org/extlink&oid=27045933&loc=d3e13503-111538

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 30
-Section 50
-Paragraph 2
-Subparagraph (a)(3)
-URI http://asc.fasb.org/extlink&oid=6379141&loc=d3e15032-111544

+ Details

Name:	us-gaap_CertainLoansAcquiredInTransferNotAccountedForAsDebtSecuritiesAcquiredDuringPeriodAtAcquisitionAtFairValue
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of credit risk valuation adjustment to derivative liabilities to properly reflect the credit quality of the entity.

+ References

No definition available.

+ Details

Name:	us-gaap_DerivativeCreditRiskValuationAdjustmentDerivativeLiabilities
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value, after offset of derivative liability, of financial asset or other contract with one or more underlyings, notional amount or payment provision or both, and the contract can be net settled by means outside the contract or delivery of an asset, before offset against an obligation to return collateral under a master netting arrangement. Includes assets elected not to be offset. Excludes assets not subject to a master netting arrangement.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 10
-URI http://asc.fasb.org/extlink&oid=51825145&loc=SL20226008-175313

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 9
-URI http://asc.fasb.org/extlink&oid=51825145&loc=SL20226006-175313

Reference 3: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 815
-SubTopic 10
-Section 50
-Paragraph 8
-Subparagraph (c)
-URI http://asc.fasb.org/extlink&oid=64837297&loc=d3e41678-113959

Reference 4: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 50
-Paragraph 3
-Subparagraph (c)
-URI http://asc.fasb.org/extlink&oid=51824906&loc=SL20225862-175312

+ Details

Name:	us-gaap_DerivativeFairValueOfDerivativeAssetAmountNotOffsetAgainstCollateral
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value, after offset of derivative asset, of financial liability or contract with one or more underlyings, notional amount or payment provision or both, and the contract can be net settled by means outside the contract or delivery of an asset, elected not to be and before offset against a right to receive collateral under a master netting arrangement. Includes liabilities elected not to be offset. Excludes liabilities not subject to a master netting arrangement.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 815
-SubTopic 10
-Section 50
-Paragraph 8
-Subparagraph (c)
-URI http://asc.fasb.org/extlink&oid=64837297&loc=d3e41678-113959

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 50
-Paragraph 3
-Subparagraph (c)
-URI http://asc.fasb.org/extlink&oid=51824906&loc=SL20225862-175312

Reference 3: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 9
-URI http://asc.fasb.org/extlink&oid=51825145&loc=SL20226006-175313

Reference 4: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 10
-URI http://asc.fasb.org/extlink&oid=51825145&loc=SL20226008-175313

+ Details

Name:	us-gaap_DerivativeFairValueOfDerivativeLiabilityAmountNotOffsetAgainstCollateral
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- Definition

Line items represent financial concepts included in a table. These concepts are used to disclose reportable information associated with domain members defined in one or many axes to the table.

+ References

No definition available.

+ Details

Name:	us-gaap_FairValueAssetsAndLiabilitiesMeasuredOnRecurringAndNonrecurringBasisLineItems
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Fair value of financial and nonfinancial obligations.

+ References

+ Details

Name:	us-gaap_LiabilitiesFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- References

No definition available.

+ Details

Name:	us-gaap_LiabilitiesFairValueDisclosureAbstract
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

The fair value of collateral received by the entity associated with securities borrowed that must be returned if the counterparty fulfills its obligations. It appears on the balance sheet as a liability and has a corresponding asset that appears on the balance sheet called Securities Received as Collateral.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 944
-SubTopic 210
-Section S99
-Paragraph 1
-Subparagraph (SX 210.7-03.15)
-URI http://asc.fasb.org/extlink&oid=6879938&loc=d3e572229-122910

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Glossary Collateral
-URI http://asc.fasb.org/extlink&oid=6507830

+ Details

Name:	us-gaap_ObligationToReturnSecuritiesReceivedAsCollateral
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- Definition

The carrying amount (fair value) as of the balance sheet date of securities held that had been pledged by counterparties under financing and lending arrangements, pursuant to which the entity has the right by agreement or custom to sell or re-pledge such securities.

+ References

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 5
-URI http://asc.fasb.org/extlink&oid=66008989&loc=d3e5086-111524

+ Details

Name:	us-gaap_SecuritiesHeldAsCollateralAtFairValue
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Details

Name:	us-gaap_MajorTypesOfDebtAndEquitySecuritiesAxis=us-gaap_MortgageBackedSecuritiesIssuedByUSGovernmentSponsoredEnterprisesMember
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_MajorTypesOfDebtAndEquitySecuritiesAxis=us-gaap_MortgageBackedSecuritiesIssuedByPrivateEnterprisesMember
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_MajorTypesOfDebtAndEquitySecuritiesAxis=us-gaap_USGovernmentSponsoredEnterprisesDebtSecuritiesMember
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_FairValueByFairValueHierarchyLevelAxis=us-gaap_FairValueInputsLevel2Member
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_FairValueByMeasurementFrequencyAxis=us-gaap_FairValueMeasurementsRecurringMember
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_MajorTypesOfDebtAndEquitySecuritiesAxis=us-gaap_LoansReceivableMember
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_FairValueByFairValueHierarchyLevelAxis=us-gaap_FairValueInputsLevel1Member
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type:

- Details

Name:	us-gaap_FairValueByFairValueHierarchyLevelAxis=us-gaap_FairValueInputsLevel3Member
Namespace Prefix:
Data Type:	na
Balance Type:
Period Type: