Collateral Positions (Details 2) $ in Thousands	Dec. 31, 2017 USD ($)
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	$ 8,164,772
Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	8,117,036
Amortized Cost	7,192,970
Accrued Interest on Pledged Assets	19,527
Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	28,161
Amortized Cost	28,881
Accrued Interest on Pledged Assets	48
Agency MBS
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	2,734,627
Agency MBS | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	2,705,754
Amortized Cost	2,724,736
Accrued Interest on Pledged Assets	7,069
Agency MBS | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	21,756
Amortized Cost	22,476
Accrued Interest on Pledged Assets	48
Legacy Non-Agency MBS
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	1,659,115
Legacy Non-Agency MBS | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	1,652,983
Amortized Cost	1,268,702
Accrued Interest on Pledged Assets	6,132
Non-Agency MBS pledged as collateral in connection with contemporaneous repurchase and reverse repurchase agreements entered into with a single counterparty	688,100
Legacy Non-Agency MBS | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
RPL/NPL MBS
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	727,111
RPL/NPL MBS | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	726,540
Amortized Cost	723,719
Accrued Interest on Pledged Assets	571
RPL/NPL MBS | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
U.S. Treasury securities
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	472,095
U.S. Treasury securities | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	472,095
Amortized Cost
Accrued Interest on Pledged Assets	0
U.S. Treasury securities | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
CRT securities
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	596,367
CRT securities | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	595,900
Amortized Cost	542,642
Accrued Interest on Pledged Assets	467
CRT securities | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
MSR relates assets
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	483,278
MSR relates assets | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	482,158
Amortized Cost	481,354
Accrued Interest on Pledged Assets	1,120
MSR relates assets | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Residential whole loans, at fair value
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	1,478,872
Residential whole loans, at fair value | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	1,474,704
Amortized Cost	1,444,915
Accrued Interest on Pledged Assets	4,168
Residential whole loans, at fair value | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Cash
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Total Fair Value of Assets Pledged and Accrued Interest	13,307
Cash | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	6,902
Amortized Cost	6,902
Accrued Interest on Pledged Assets	0
Cash | Derivative Hedging Instruments
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	6,405
Amortized Cost	6,405
Accrued Interest on Pledged Assets	0
Fair Value, Estimate Not Practicable, Carrying (Reported) Amount | Residential whole loans, at fair value | Assets Pledged Under Repurchase Agreements
Transfer of Certain Financial Assets Accounted for as Secured Borrowings [Line Items]
Fair Value	478,500
Amortized Cost	448,700
Total Fair Value of Assets Pledged and Accrued Interest	$ 996,200