Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($)	Jun. 30, 2016	Dec. 31, 2015
Assets:
Available-for-sale securities	$ 10,685,331,000	$ 11,356,643,000
Securities obtained and pledged as collateral	512,059,000	507,443,000
Residential whole loans, at fair value	684,582,000	623,276,000
Swaps	0	1,127,000
Liabilities:
Obligation to return securities obtained as collateral	512,059,000	507,443,000
Agency MBS
Assets:
Available-for-sale securities	4,307,882,000	4,752,244,000
Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,104,880,000	6,420,817,000
CRT securities
Assets:
Available-for-sale securities	272,569,000	183,582,000
Recurring basis
Assets:
Securities obtained and pledged as collateral	512,059,000	507,443,000
Swaps		1,127,000
Total assets carried at fair value	11,881,972,000	12,488,489,000
Liabilities:
Swaps	131,971,000	70,526,000
Obligation to return securities obtained as collateral	512,059,000	507,443,000
Total liabilities carried at fair value	644,030,000	577,969,000
Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,307,882,000	4,752,244,000
Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,104,880,000	6,420,817,000
Recurring basis | CRT securities
Assets:
Available-for-sale securities	272,569,000	183,582,000
Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	684,582,000	623,276,000
Level 1 | Recurring basis
Assets:
Securities obtained and pledged as collateral	512,059,000	507,443,000
Swaps		0
Total assets carried at fair value	512,059,000	507,443,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	512,059,000	507,443,000
Total liabilities carried at fair value	512,059,000	507,443,000
Level 1 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 2
Components of financial instruments carried at fair value
Credit valuation adjustment to derivative liabilities	0
Credit valuation adjustment to derivative assets	0
Level 2 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps		1,127,000
Total assets carried at fair value	10,685,331,000	11,357,770,000
Liabilities:
Swaps	131,971,000	70,526,000
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	131,971,000	70,526,000
Level 2 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	4,307,882,000	4,752,244,000
Level 2 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	6,104,880,000	6,420,817,000
Level 2 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	272,569,000	183,582,000
Level 2 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	0	0
Level 3 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps		0
Total assets carried at fair value	684,582,000	623,276,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	0
Level 3 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Non-Agency MBS, including MBS transferred to consolidated VIEs
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Residential Whole Loans
Assets:
Residential whole loans, at fair value	$ 684,582,000	$ 623,276,000