Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value (Details) - USD ($) $ in Thousands		Dec. 31, 2015		Dec. 31, 2014
Financial Assets:
MBS and CRT securities		$ 11,356,643		$ 10,762,622
Securities obtained and pledged as collateral		507,443		512,105
Residential whole loans, at carrying value		271,845		207,923
Residential whole loans, at fair value		623,276		143,472
Restricted cash		71,538		67,255
Linked Transactions		0		398,336
Swaps		1,127		3,136
Financial Liabilities:
Securitized debt	[1]	22,057		110,574
Obligation to return securities obtained as collateral		507,443		512,105
Senior Notes		100,000		100,000
Carrying value
Financial Assets:
Securities obtained and pledged as collateral		507,443		512,105
Cash and cash equivalents		165,007		182,437
Restricted cash		71,538		67,255
Linked Transactions		0		398,336
Swaps		1,127		3,136
Financial Liabilities:
Repurchase agreements		7,888,902		8,267,388
FHLB advances		1,500,000		0
Securitized debt		22,057		110,574
Obligation to return securities obtained as collateral		507,443		512,105
Senior Notes		100,000		100,000
Swaps		70,526		62,198
Estimated fair value
Financial Assets:
Securities obtained and pledged as collateral		507,443		512,105
Cash and cash equivalents		165,007		182,437
Restricted cash		71,538		67,255
Linked Transactions		0		398,336
Swaps		1,127		3,136
Financial Liabilities:
Repurchase agreements		7,828,115		8,266,699
FHLB advances		1,500,000		0
Securitized debt		22,057		110,791
Obligation to return securities obtained as collateral		507,443		512,105
Senior Notes		99,391		103,031
Swaps		70,526		62,198
Agency MBS
Financial Assets:
MBS and CRT securities		4,752,244		5,904,207
Agency MBS | Carrying value
Financial Assets:
MBS and CRT securities		4,752,244		5,904,207
Agency MBS | Estimated fair value
Financial Assets:
MBS and CRT securities		4,752,244		5,904,207
Non-Agency MBS
Financial Assets:
MBS and CRT securities	[2]	6,420,817		4,755,432
Non-Agency MBS | Carrying value
Financial Assets:
MBS and CRT securities		6,420,817		4,755,432
Non-Agency MBS | Estimated fair value
Financial Assets:
MBS and CRT securities		6,420,817		4,755,432
CRT securities
Financial Assets:
MBS and CRT securities		183,582	[3]	102,983
CRT securities | Carrying value
Financial Assets:
MBS and CRT securities		183,582		102,983
CRT securities | Estimated fair value
Financial Assets:
MBS and CRT securities		183,582		102,983
Residential whole loans | Carrying value
Financial Assets:
Residential whole loans, at carrying value		271,845		207,923
Residential whole loans, at fair value		623,276		143,472
Residential whole loans | Estimated fair value
Financial Assets:
Residential whole loans, at carrying value		289,696		217,386
Residential whole loans, at fair value		$ 623,276		$ 143,472

[1] Securitized Debt represents third-party liabilities of consolidated VIEs and excludes liabilities of the VIEs acquired by the Company that eliminate on consolidation. The third-party beneficial interest holders in the VIEs have no recourse to the general credit of the Company. (See Notes 12 and 17 for further discussion.)

[2] At December 31, 2015 and 2014, the Company expected to recover approximately 89% and 83%, respectively, of the then-current face amount of Non-Agency MBS.

[3] Amounts disclosed at December 31, 2015 includes CRT securities with a fair value of $62.2 million for which the fair value option has been elected. Such securities have gross unrealized gains of approximately $332,000, gross unrealized losses of approximately $555,000 and net unrealized losses of approximately $223,000 at December 31, 2015.