Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($)	Mar. 31, 2016	Dec. 31, 2015
Assets:
Available-for-sale securities	$ 10,861,443,000	$ 11,356,643,000
Securities obtained and pledged as collateral	513,409,000	507,443,000
Residential whole loans, at fair value	647,360,000	623,276,000
Swaps	152,000	1,127,000
Liabilities:
Obligation to return securities obtained as collateral	513,409,000	507,443,000
Agency MBS
Assets:
Available-for-sale securities	4,544,883,000	4,752,244,000
Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,100,757,000	6,420,817,000
CRT securities
Assets:
Available-for-sale securities	215,803,000	183,582,000
Recurring basis
Assets:
Securities obtained and pledged as collateral	513,409,000	507,443,000
Swaps	152,000	1,127,000
Total assets carried at fair value	12,022,364,000	12,488,489,000
Liabilities:
Swaps	118,893,000	70,526,000
Obligation to return securities obtained as collateral	513,409,000	507,443,000
Total liabilities carried at fair value	632,302,000	577,969,000
Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,544,883,000	4,752,244,000
Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,100,757,000	6,420,817,000
Recurring basis | CRT securities
Assets:
Available-for-sale securities	215,803,000	183,582,000
Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	647,360,000	623,276,000
Level 1 | Recurring basis
Assets:
Securities obtained and pledged as collateral	513,409,000	507,443,000
Swaps	0	0
Total assets carried at fair value	513,409,000	507,443,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	513,409,000	507,443,000
Total liabilities carried at fair value	513,409,000	507,443,000
Level 1 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 2
Components of financial instruments carried at fair value
Credit valuation adjustment to derivative liabilities	0
Credit valuation adjustment to derivative assets	0
Level 2 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	152,000	1,127,000
Total assets carried at fair value	10,861,595,000	11,357,770,000
Liabilities:
Swaps	118,893,000	70,526,000
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	118,893,000	70,526,000
Level 2 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,544,883,000	4,752,244,000
Level 2 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,100,757,000	6,420,817,000
Level 2 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	215,803,000	183,582,000
Level 2 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 3 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	0	0
Total assets carried at fair value	647,360,000	623,276,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	0
Level 3 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	$ 647,360,000	$ 623,276,000