Fair Value of Financial Instruments - Carrying Value vs Fair Value (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Financial Assets:
Available-for-sale securities	$ 9,460,859	$ 9,969,163
Securities obtained and pledged as collateral	371,333	510,767
Residential whole loans, at carrying value	573,715	590,540
Residential whole loans, at fair value	775,152	814,682
Restricted cash	10,980	58,463
MSR related investments held at fair value	93,002	85,800
Swaps	119	233
Financial Liabilities (1):
Obligation to return securities obtained as collateral	510,733	510,767
Senior Notes	96,743	96,733
Carrying Value
Financial Assets:
Securities obtained and pledged as collateral	371,333	510,767
Cash and cash equivalents	421,572	260,112
Restricted cash	10,980	58,463
MSR related investments held at fair value	93,002	85,800
Swaps	119	233
Financial Liabilities (1):
Repurchase agreements	8,137,102	8,472,268
FHLB advances	0	215,000
Obligation to return securities obtained as collateral	510,733	510,767
Senior Notes	96,743	96,733
Swaps	0	46,954
Estimated Fair Value
Financial Assets:
Securities obtained and pledged as collateral	371,333	510,767
Cash and cash equivalents	421,572	260,112
Restricted cash	10,980	58,463
MSR related investments held at fair value	93,002	85,800
Swaps	119	233
Financial Liabilities (1):
Repurchase agreements	8,137,251	8,472,078
FHLB advances	0	215,000
Obligation to return securities obtained as collateral	510,733	510,767
Senior Notes	100,591	101,111
Swaps	0	46,954
Agency MBS
Financial Assets:
Available-for-sale securities	3,494,614	3,738,497
Agency MBS | Carrying Value
Financial Assets:
Available-for-sale securities	3,494,614	3,738,497
Agency MBS | Estimated Fair Value
Financial Assets:
Available-for-sale securities	3,494,614	3,738,497
Non-Agency MBS
Financial Assets:
Available-for-sale securities	5,468,178	5,825,816
Non-Agency MBS | Carrying Value
Financial Assets:
Available-for-sale securities	5,468,178	5,825,816
Non-Agency MBS | Estimated Fair Value
Financial Assets:
Available-for-sale securities	5,468,178	5,825,816
CRT securities
Financial Assets:
Available-for-sale securities	498,067	404,850
CRT securities | Carrying Value
Financial Assets:
Available-for-sale securities	498,067	404,850
CRT securities | Estimated Fair Value
Financial Assets:
Available-for-sale securities	498,067	404,850
Residential whole loans | Carrying Value
Financial Assets:
Residential whole loans, at carrying value	573,715	590,540
Residential whole loans, at fair value	775,152	814,682
Residential whole loans | Estimated Fair Value
Financial Assets:
Residential whole loans, at carrying value	608,209	621,548
Residential whole loans, at fair value	$ 775,152	$ 814,682