Fair Value of Financial Instruments - Carry Value vs Fair Value (Details) - USD ($) $ in Thousands		Dec. 31, 2017	Dec. 31, 2016
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		$ 7,023,050	$ 9,828,183
MSR related assets		492,080	226,780
Residential whole loans, at carrying value	[1]	908,516	590,540
Residential whole loans, at fair value	[1]	1,325,115	814,682
Securities obtained and pledged as collateral, at fair value		504,062	510,767
Restricted cash		13,307	58,463
Swaps, at fair value		679	233
Obligation to return securities obtained as collateral, at fair value		504,062	510,767
Securitized debt		363,944	0
Senior Notes		96,773	96,733
Carrying value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
MSR related assets		492,080	226,780
Securities obtained and pledged as collateral, at fair value		504,062	510,767
Cash and cash equivalents		449,757	260,112
Restricted cash		13,307	58,463
Swaps, at fair value		679	233
Repurchase agreements		6,614,701	8,472,268
FHLB advances		0	215,000
Obligation to return securities obtained as collateral, at fair value		504,062	510,767
Securitized debt		363,944	0
Senior Notes		96,773	96,733
Swaps		0	46,954
Estimated fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
MSR related assets		493,026	226,780
Securities obtained and pledged as collateral, at fair value		504,062	510,767
Cash and cash equivalents		449,757	260,112
Restricted cash		13,307	58,463
Swaps, at fair value		679	233
Repurchase agreements		6,623,255	8,472,078
FHLB advances		0	215,000
Obligation to return securities obtained as collateral, at fair value		504,062	510,767
Securitized debt		366,109	0
Senior Notes		103,729	101,111
Swaps		0	46,954
Agency MBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		2,824,681	3,738,497
Agency MBS | Carrying value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		2,824,681	3,738,497
Agency MBS | Estimated fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		2,824,681	3,738,497
Non-Agency MBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		3,533,966	5,684,836
Non-Agency MBS | Carrying value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		3,533,966	5,684,836
Non-Agency MBS | Estimated fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		3,533,966	5,684,836
CRT securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		664,403	404,850
CRT securities | Carrying value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		664,403	404,850
CRT securities | Estimated fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Available-for-sale Securities		664,403	404,850
Residential whole loans, at fair value | Carrying value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential whole loans, at carrying value		908,516	590,540
Residential whole loans, at fair value		1,325,115	814,682
Residential whole loans, at fair value | Estimated fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential whole loans, at carrying value		988,688	621,548
Residential whole loans, at fair value		$ 1,325,115	$ 814,682

[1] Includes approximately $183.2 million of Residential whole loans, at carrying value and $289.3 million of Residential whole loans, at fair value transferred to consolidated VIEs at December 31, 2017. Such assets can be used only to settle the obligations of each respective VIE.