Fair Value of Financial Instruments-Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands		Sep. 30, 2015	Dec. 31, 2014
Financial Assets:
Available-for-sale securities		$ 11,694,667	$ 10,762,622
Securities obtained and pledged as collateral		508,838	512,105
Residential whole loans, at carrying value		245,894	207,923
Residential whole loans, at fair value		531,537	143,472
Restricted cash		109,997	67,255
Linked Transactions		0	398,336
Swaps		18	3,136
Financial Liabilities:
Securitized debt	[1]	32,217	110,574
Obligation to return securities obtained as collateral		509,620	512,105
Senior Notes		100,000	100,000
Carrying Value
Financial Assets:
Securities obtained and pledged as collateral		509,620	512,105
Cash and cash equivalents		174,160	182,437
Restricted cash		109,997	67,255
Linked Transactions		0	398,336
Swaps		18	3,136
Financial Liabilities:
Repurchase agreements		9,210,834	8,267,388
FHLB advances		265,000	0
Securitized debt		32,217	110,574
Obligation to return securities obtained as collateral		509,620	512,105
Senior Notes		100,000	100,000
Swaps		105,473	62,198
Estimated Fair Value
Financial Assets:
Securities obtained and pledged as collateral		509,620	512,105
Cash and cash equivalents		174,160	182,437
Restricted cash		109,997	67,255
Linked Transactions		0	398,336
Swaps		18	3,136
Financial Liabilities:
Repurchase agreements		9,210,989	8,266,699
FHLB advances		265,000	0
Securitized debt		32,298	110,791
Obligation to return securities obtained as collateral		509,620	512,105
Senior Notes		100,351	103,031
Swaps		105,473	62,198
Agency MBS
Financial Assets:
Available-for-sale securities		5,020,477	5,904,207
Agency MBS | Carrying Value
Financial Assets:
Available-for-sale securities		5,020,477	5,904,207
Agency MBS | Estimated Fair Value
Financial Assets:
Available-for-sale securities		5,020,477	5,904,207
Non-Agency MBS, including MBS transferred to consolidated VIEs
Financial Assets:
Available-for-sale securities		6,524,222	4,755,432
Non-Agency MBS, including MBS transferred to consolidated VIEs | Carrying Value
Financial Assets:
Available-for-sale securities		6,524,222	4,755,432
Non-Agency MBS, including MBS transferred to consolidated VIEs | Estimated Fair Value
Financial Assets:
Available-for-sale securities		6,524,222	4,755,432
CRT securities
Financial Assets:
Available-for-sale securities		149,968	102,983
CRT securities | Carrying Value
Financial Assets:
Available-for-sale securities		149,968	102,983
CRT securities | Estimated Fair Value
Financial Assets:
Available-for-sale securities		149,968	102,983
Residential whole loans | Carrying Value
Financial Assets:
Residential whole loans, at carrying value		245,894	207,923
Residential whole loans, at fair value		531,537	143,472
Residential whole loans | Estimated Fair Value
Financial Assets:
Residential whole loans, at carrying value		262,330	217,386
Residential whole loans, at fair value		$ 531,537	$ 143,472

[1] Securitized debt represents third-party liabilities of consolidated VIEs and excludes liabilities of the VIEs acquired by the Company that eliminate on consolidation. The third-party beneficial interest holders in the VIEs have no recourse to the general credit of the Company. (See Notes 12 and 17 for further discussion.)