Collateral Positions (Details 2) $ in Thousands	Jun. 30, 2018 USD ($)
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	$ 7,257,501
Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	2,292,698
Legacy Non-Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,820,478
RPL/NPL MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	634,575
U.S. Treasury securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	220,731
CRT securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	516,937
MSR related assets
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	381,818
Residential whole loans
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,386,966
Cash
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	3,298
Repurchase Agreements and Other Advances
Collateral Positions
Fair Value	7,234,717
Amortized Cost	6,537,924
Accrued Interest on Pledged Assets	19,679
Repurchase Agreements and Other Advances | Agency MBS
Collateral Positions
Fair Value	2,283,312
Amortized Cost	2,318,975
Accrued Interest on Pledged Assets	6,281
Repurchase Agreements and Other Advances | Legacy Non-Agency MBS
Collateral Positions
Fair Value	1,813,359
Amortized Cost	1,377,319
Accrued Interest on Pledged Assets	7,119
Non-Agency MBS pledged as collateral in connection with contemporaneous repurchase and reverse repurchase agreements entered into with a single counterparty	320,300
Repurchase Agreements and Other Advances | RPL/NPL MBS
Collateral Positions
Fair Value	634,073
Amortized Cost	633,223
Accrued Interest on Pledged Assets	502
Repurchase Agreements and Other Advances | U.S. Treasury securities
Collateral Positions
Fair Value	220,731
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Repurchase Agreements and Other Advances | CRT securities
Collateral Positions
Fair Value	516,486
Amortized Cost	475,568
Accrued Interest on Pledged Assets	451
Repurchase Agreements and Other Advances | MSR related assets
Collateral Positions
Fair Value	381,390
Amortized Cost	380,350
Accrued Interest on Pledged Assets	428
Repurchase Agreements and Other Advances | Residential whole loans
Collateral Positions
Fair Value	1,382,068
Amortized Cost	1,349,191
Accrued Interest on Pledged Assets	4,898
Repurchase Agreements and Other Advances | Cash
Collateral Positions
Fair Value	3,298
Amortized Cost	3,298
Accrued Interest on Pledged Assets	0
Repurchase Agreements and Other Advances | Residential Loans At Fair Value
Collateral Positions
Fair Value	429,700
Amortized Cost	396,900
Total Fair Value of Assets Pledged and Accrued Interest	952,300
Derivative Hedging Instruments
Collateral Positions
Fair Value	3,097
Amortized Cost	3,242
Accrued Interest on Pledged Assets	8
Derivative Hedging Instruments | Agency MBS
Collateral Positions
Fair Value	3,097
Amortized Cost	3,242
Accrued Interest on Pledged Assets	8
Derivative Hedging Instruments | Legacy Non-Agency MBS
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | RPL/NPL MBS
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | U.S. Treasury securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | CRT securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | MSR related assets
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Residential whole loans
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Cash
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	$ 0