Fair Value of Financial Instruments (Details 2) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014		Dec. 31, 2013
Financial Assets:
Available-for-sale Securities	$ 10,937,502		$ 11,371,358
Securities obtained and pledged as collateral	442,370		383,743
Restricted cash	43,751		37,520
Linked Transactions	190,681		28,181
Swaps	4,322		13,000
Financial Liabilities:
Repurchase agreements	8,125,723	[1]	8,339,297	[1]
Securitized debt	156,276	[2]	366,205	[2]
Obligation to return securities obtained as collateral	442,370		383,743
Senior Notes	100,000		100,000
Swaps	35,493	[3]	28,217	[3]
Carrying Value
Financial Assets:
Securities obtained and pledged as collateral	442,370		383,743
Cash and cash equivalents	423,891		565,370
Restricted cash	43,751		37,520
Linked Transactions	190,681		28,181
Swaps	4,322		13,000
Residential whole loans	112,924		0
Financial Liabilities:
Repurchase agreements	8,125,723		8,339,297
Securitized debt	156,276		366,205
Obligation to return securities obtained as collateral	442,370		383,743
Senior Notes	100,000		100,000
Swaps	35,493		28,217
Estimated Fair Value
Financial Assets:
Securities obtained and pledged as collateral	442,370		383,743
Cash and cash equivalents	423,891		565,370
Restricted cash	43,751		37,520
Linked Transactions	190,681		28,181
Swaps	4,322		13,000
Residential whole loans	114,357		0
Financial Liabilities:
Repurchase agreements	8,124,355		8,339,071
Securitized debt	157,264		366,767
Obligation to return securities obtained as collateral	442,370		383,743
Senior Notes	105,000		98,000
Swaps	35,493		28,217
Agency Mortgage Backed Securities
Financial Assets:
Available-for-sale Securities	6,174,176		6,519,221
Agency Mortgage Backed Securities | Carrying Value
Financial Assets:
Available-for-sale Securities	6,174,176		6,519,221
Agency Mortgage Backed Securities | Estimated Fair Value
Financial Assets:
Available-for-sale Securities	6,174,176		6,519,221
Non-Agency Mortgage Backed Securities
Financial Assets:
Available-for-sale Securities	4,763,326		4,852,137
Non-Agency Mortgage Backed Securities | Carrying Value
Financial Assets:
Available-for-sale Securities	4,763,326		4,852,137
Non-Agency Mortgage Backed Securities | Estimated Fair Value
Financial Assets:
Available-for-sale Securities	$ 4,763,326		$ 4,852,137

[1] The fair value of securities pledged against the Company’s repurchase agreements was $9.595 billion and $10.116 billion at September 30, 2014 and December 31, 2013, respectively.

[2] Securitized Debt represents third-party liabilities of consolidated VIEs and excludes liabilities of the VIEs acquired by the Company that eliminate on consolidation. The third-party beneficial interest holders in the VIEs have no recourse to the general credit of the Company. (See Notes 10 and 15 for further discussion.)

[3] The fair value of securities pledged against the Company’s Swaps was $67.4 million and $73.9 million at September 30, 2014 and December 31, 2013, respectively.