Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($)	Sep. 30, 2016	Dec. 31, 2015
Assets:
Available-for-sale securities	$ 10,277,990,000	$ 11,356,643,000
Securities obtained and pledged as collateral	506,182,000	507,443,000
Residential whole loans, at fair value	797,191,000	623,276,000
Swaps	121,000	1,127,000
Liabilities:
Obligation to return securities obtained as collateral	506,182,000	507,443,000
Agency MBS
Assets:
Available-for-sale securities	4,021,535,000	4,752,244,000
Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	5,908,815,000	6,420,817,000
CRT securities
Assets:
Available-for-sale securities	347,640,000	183,582,000
Recurring basis
Assets:
Securities obtained and pledged as collateral	506,182,000	507,443,000
Swaps	121,000	1,127,000
Total assets carried at fair value	11,581,484,000	12,488,489,000
Liabilities:
Swaps	109,323,000	70,526,000
Obligation to return securities obtained as collateral	506,182,000	507,443,000
Total liabilities carried at fair value	615,505,000	577,969,000
Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,021,535,000	4,752,244,000
Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	5,908,815,000	6,420,817,000
Recurring basis | CRT securities
Assets:
Available-for-sale securities	347,640,000	183,582,000
Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	797,191,000	623,276,000
Level 1 | Recurring basis
Assets:
Securities obtained and pledged as collateral	506,182,000	507,443,000
Swaps	0	0
Total assets carried at fair value	506,182,000	507,443,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	506,182,000	507,443,000
Total liabilities carried at fair value	506,182,000	507,443,000
Level 1 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 2
Components of financial instruments carried at fair value
Credit valuation adjustment to derivative liabilities	0
Credit valuation adjustment to derivative assets	0
Level 2 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	121,000	1,127,000
Total assets carried at fair value	10,278,111,000	11,357,770,000
Liabilities:
Swaps	109,323,000	70,526,000
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	109,323,000	70,526,000
Level 2 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,021,535,000	4,752,244,000
Level 2 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	5,908,815,000	6,420,817,000
Level 2 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	347,640,000	183,582,000
Level 2 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 3 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	0	0
Total assets carried at fair value	797,191,000	623,276,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	0
Level 3 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	$ 797,191,000	$ 623,276,000