Repurchase Agreements and Other Advances (Details) - USD ($) $ in Thousands		12 Months Ended
		Dec. 31, 2015	Dec. 31, 2014
Repurchase agreements
Repurchase Agreements
Weighted average remaining term-to-interest rate reset of borrowings under repurchase agreements		21 days	25 days
Effective repricing period		18 months	21 months
Balance of repurchase agreements		$ 7,888,902	$ 8,267,388	[1]
Weighted Average Interest Rate		1.48%	0.79%
Repurchase agreements | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Weighted Average Interest Rate		0.00%
Repurchase agreements | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		$ 5,742,349
Weighted Average Interest Rate		1.27%
Repurchase agreements | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		$ 938,181
Weighted Average Interest Rate		1.71%
Repurchase agreements | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		$ 1,014,263
Weighted Average Interest Rate		2.19%
Repurchase agreements | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		$ 194,109
Weighted Average Interest Rate		3.11%
Repurchase agreements | Interest Rate Reset within 30 Days
Repurchase Agreements
Balance of repurchase agreements		$ 7,054,483	$ 7,144,737	[1]
Weighted Average Interest Rate		1.44%	0.72%
Repurchase agreements | Interest Rate Reset Over 30 Days to 3 Months
Repurchase Agreements
Balance of repurchase agreements		$ 734,955	$ 1,000,313	[1]
Weighted Average Interest Rate		1.79%	1.12%
Repurchase agreements | Interest Rate Reset Over 6 Months to 12 Months
Repurchase Agreements
Balance of repurchase agreements		$ 99,464	$ 122,338	[1]
Weighted Average Interest Rate		2.36%	1.98%
Repurchase agreements | Agency MBS
Repurchase Agreements
Balance of repurchase agreements		$ 2,727,542	$ 5,177,835
Fair Value of assets pledged as collateral under repurchase agreements		$ 2,881,049	$ 5,462,566
Weighted average haircut (as a percent)	[2]	4.67%	4.79%
Repurchase agreements | Agency MBS | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Repurchase agreements | Agency MBS | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		2,563,741
Repurchase agreements | Agency MBS | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		163,801
Repurchase agreements | Agency MBS | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Agency MBS | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Legacy Non-Agency MBS
Repurchase Agreements
Balance of repurchase agreements	[3]	1,960,222	$ 2,233,236
Fair Value of assets pledged as collateral under repurchase agreements	[3],[4]	$ 2,818,968	$ 3,491,312
Weighted average haircut (as a percent)	[2]	25.42%	28.88%
Repurchase agreements | RPL/NPL MBS
Repurchase Agreements
Balance of repurchase agreements	[3]	$ 2,080,163	$ 130,919
Fair Value of assets pledged as collateral under repurchase agreements	[3]	$ 2,625,866	$ 160,688
Weighted average haircut (as a percent)	[2]	21.37%	20.00%
Repurchase agreements | RPL/NPL MBS | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Repurchase agreements | RPL/NPL MBS | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		1,670,586
Repurchase agreements | RPL/NPL MBS | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		143,705
Repurchase agreements | RPL/NPL MBS | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		265,872
Repurchase agreements | RPL/NPL MBS | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Non-Agency MBS
Repurchase Agreements
Balance of repurchase agreements		1,960,222
Repurchase agreements | Non-Agency MBS | Overnight
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Non-Agency MBS | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		892,341
Repurchase agreements | Non-Agency MBS | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		613,131
Repurchase agreements | Non-Agency MBS | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		454,750
Repurchase agreements | Non-Agency MBS | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | U.S. Treasuries
Repurchase Agreements
Balance of repurchase agreements		504,760	$ 507,114
Fair Value of assets pledged as collateral under repurchase agreements		$ 507,443	$ 512,105
Weighted average haircut (as a percent)	[2]	1.60%	1.62%
Repurchase agreements | U.S. Treasuries | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Repurchase agreements | U.S. Treasuries | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		504,760
Repurchase agreements | U.S. Treasuries | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | U.S. Treasuries | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | U.S. Treasuries | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | CRT securities
Repurchase Agreements
Balance of repurchase agreements	[3]	128,465	$ 75,960
Fair Value of assets pledged as collateral under repurchase agreements	[3]	$ 170,352	$ 94,610
Weighted average haircut (as a percent)	[2]	25.04%	25.00%
Repurchase agreements | CRT securities | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Repurchase agreements | CRT securities | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		110,921
Repurchase agreements | CRT securities | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		17,544
Repurchase agreements | CRT securities | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | CRT securities | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Residential whole loans
Repurchase Agreements
Balance of repurchase agreements		487,750	$ 142,324
Fair Value of assets pledged as collateral under repurchase agreements		$ 684,136	$ 212,986
Weighted average haircut (as a percent)	[2]	27.69%	33.43%
Repurchase agreements | Residential whole loans | Overnight
Repurchase Agreements
Balance of repurchase agreements		$ 0
Repurchase agreements | Residential whole loans | Within 30 days
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Residential whole loans | Over 30 days to 90 days
Repurchase Agreements
Balance of repurchase agreements		0
Repurchase agreements | Residential whole loans | Over 90 days to 12 months
Repurchase Agreements
Balance of repurchase agreements		293,641
Repurchase agreements | Residential whole loans | Over 12 Months
Repurchase Agreements
Balance of repurchase agreements		194,109
Repurchase agreements | Non-Agency MBS
Repurchase Agreements
Non-Agency MBS acquired from consolidated VIEs		$ 570,500	$ 1,275,000
Linked Transactions | Non-Agency MBS
Repurchase Agreements
Linked repurchase agreements			$ 1,520,000

[1] At December 31, 2014, the Company had repurchase agreements of $1.520 billion that were linked to securities purchased and accounted for as Linked Transactions, and as such, the linked repurchase agreements are not included in the above table. As previously discussed, new accounting guidance effective January 1, 2015 prospectively eliminated the use of Linked Transaction accounting. (See Note 6)

[2] Haircut represents the percentage amount by which the collateral value is contractually required to exceed the loan amount.

[3] Does not reflect Legacy Non-Agency MBS, RPL/NPL MBS, CRT securities and repurchase agreement borrowings that were components of Linked Transactions at December 31, 2014. As previously discussed, new accounting guidance effective January 1, 2015 prospectively eliminated the use of Linked Transaction accounting. (See Note 6)

[4] Includes $570.5 million and $1.275 billion of Legacy Non-Agency MBS acquired from consolidated VIEs at December 31, 2015 and 2014, respectively, that are eliminated from the Company’s consolidated balance sheets.