Fair Value of Financial Instruments - Fair Value Hierarchy (Details) - USD ($)	Mar. 31, 2017	Dec. 31, 2016
Assets:
Available-for-sale securities	$ 9,460,859,000	$ 9,969,163,000
Securities obtained and pledged as collateral	371,333,000	510,767,000
Residential whole loans, at fair value	775,152,000	814,682,000
MSR related investments held at fair value	93,002,000	85,800,000
Swaps	119,000	233,000
Liabilities:
Obligation to return securities obtained as collateral	510,733,000	510,767,000
Agency MBS
Assets:
Available-for-sale securities	3,494,614,000	3,738,497,000
Non-Agency MBS
Assets:
Available-for-sale securities	5,468,178,000	5,825,816,000
CRT securities
Assets:
Available-for-sale securities	498,067,000	404,850,000
Recurring basis
Assets:
Securities obtained and pledged as collateral	371,333,000	510,767,000
Swaps	119,000	233,000
Total assets carried at fair value	10,607,463,000	11,294,845,000
Liabilities:
Swaps	0	46,954,000
Obligation to return securities obtained as collateral	510,733,000	510,767,000
Total liabilities carried at fair value	510,733,000	557,721,000
Recurring basis | Agency MBS
Assets:
Available-for-sale securities	3,494,614,000	3,738,497,000
Recurring basis | Non-Agency MBS
Assets:
Available-for-sale securities	5,468,178,000	5,825,816,000
Recurring basis | CRT securities
Assets:
Available-for-sale securities	498,067,000	404,850,000
Recurring basis | Residential whole loans
Assets:
Residential whole loans, at fair value	775,152,000	814,682,000
Level 1 | Recurring basis
Assets:
Securities obtained and pledged as collateral	371,333,000	510,767,000
Swaps	0	0
Total assets carried at fair value	371,333,000	510,767,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	510,733,000	510,767,000
Total liabilities carried at fair value	510,733,000	510,767,000
Level 1 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Non-Agency MBS
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 1 | Recurring basis | Residential whole loans
Assets:
Residential whole loans, at fair value	0	0
Level 2
Components of financial instruments carried at fair value
Credit valuation adjustment to derivative liabilities	0
Credit valuation adjustment to derivative assets	0
Level 2 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	119,000	233,000
Total assets carried at fair value	9,178,646,000	9,969,396,000
Liabilities:
Swaps	0	46,954,000
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	46,954,000
Level 2 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	3,494,614,000	3,738,497,000
Level 2 | Recurring basis | Non-Agency MBS
Assets:
Available-for-sale securities	5,185,846,000	5,825,816,000
Level 2 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	498,067,000	404,850,000
Level 2 | Recurring basis | Residential whole loans
Assets:
Residential whole loans, at fair value	0	0
Level 3 | Recurring basis
Assets:
Securities obtained and pledged as collateral	0	0
Swaps	0	0
Total assets carried at fair value	1,057,484,000	814,682,000
Liabilities:
Swaps	0	0
Obligation to return securities obtained as collateral	0	0
Total liabilities carried at fair value	0	0
Level 3 | Recurring basis | Agency MBS
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Non-Agency MBS
Assets:
Available-for-sale securities	282,332,000	0
Level 3 | Recurring basis | CRT securities
Assets:
Available-for-sale securities	0	0
Level 3 | Recurring basis | Residential whole loans
Assets:
Residential whole loans, at fair value	$ 775,152,000	$ 814,682,000