Fair Value of Financial Instruments-Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands		Jun. 30, 2015	Dec. 31, 2014
Financial Assets:
Available-for-sale securities		$ 12,342,274	$ 10,762,622
Securities obtained and pledged as collateral		498,336	512,105
Residential whole loans, at carrying value		245,402	207,923
Residential whole loans, at fair value		183,861	143,472
Restricted cash		68,057	67,255
Linked Transactions		0	398,336
Swaps		849	3,136
Financial Liabilities:
Repurchase agreements		9,635,036	8,267,388
Securitized debt	[1]	62,320	110,574
Obligation to return securities obtained as collateral		498,336	512,105
Senior Notes		100,000	100,000
Carrying Value
Financial Assets:
Securities obtained and pledged as collateral		498,336	512,105
Cash and cash equivalents		218,492	182,437
Restricted cash		68,057	67,255
Linked Transactions		0	398,336
Swaps		849	3,136
Financial Liabilities:
Repurchase agreements		9,635,036	8,267,388
Securitized debt		62,320	110,574
Obligation to return securities obtained as collateral		498,336	512,105
Senior Notes		100,000	100,000
Swaps		65,420	62,198
Estimated Fair Value
Financial Assets:
Securities obtained and pledged as collateral		498,336	512,105
Cash and cash equivalents		218,492	182,437
Restricted cash		68,057	67,255
Linked Transactions		0	398,336
Swaps		849	3,136
Financial Liabilities:
Repurchase agreements		9,638,056	8,266,699
Securitized debt		62,558	110,791
Obligation to return securities obtained as collateral		498,336	512,105
Senior Notes		104,600	104,720
Swaps		65,420	62,198
Agency MBS
Financial Assets:
Available-for-sale securities		5,331,211	5,904,207
Agency MBS | Carrying Value
Financial Assets:
Available-for-sale securities		5,331,211	5,904,207
Agency MBS | Estimated Fair Value
Financial Assets:
Available-for-sale securities		5,331,211	5,904,207
Non-Agency MBS, including MBS transferred to consolidated VIEs
Financial Assets:
Available-for-sale securities		6,882,153	4,755,432
Non-Agency MBS, including MBS transferred to consolidated VIEs | Carrying Value
Financial Assets:
Available-for-sale securities		6,882,153	4,755,432
Non-Agency MBS, including MBS transferred to consolidated VIEs | Estimated Fair Value
Financial Assets:
Available-for-sale securities		6,882,153	4,755,432
CRT securities
Financial Assets:
Available-for-sale securities		128,910	102,983
CRT securities | Carrying Value
Financial Assets:
Available-for-sale securities		128,910	102,983
CRT securities | Estimated Fair Value
Financial Assets:
Available-for-sale securities		128,910	102,983
Residential whole loans | Carrying Value
Financial Assets:
Residential whole loans, at carrying value		245,402	207,923
Residential whole loans, at fair value		183,861	143,472
Residential whole loans | Estimated Fair Value
Financial Assets:
Residential whole loans, at carrying value		261,925	217,386
Residential whole loans, at fair value		$ 183,861	$ 143,472

[1] Securitized debt represents third-party liabilities of consolidated VIEs and excludes liabilities of the VIEs acquired by the Company that eliminate on consolidation. The third-party beneficial interest holders in the VIEs have no recourse to the general credit of the Company. (See Notes 12 and 17 for further discussion.)