Collateral Positions (Details 2) $ in Thousands	Mar. 31, 2018 USD ($)
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	$ 8,157,133
Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	2,554,077
Legacy Non-Agency MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,904,284
RPL/NPL MBS
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	733,985
U.S. Treasury securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	220,814
CRT securities
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	591,039
MSR related assets
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	433,552
Residential whole loans
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	1,712,282
Cash
Collateral Positions
Total Fair Value of Assets Pledged and Accrued Interest	7,100
Repurchase Agreements and Other Advances
Collateral Positions
Fair Value	8,119,091
Amortized Cost	7,399,703
Accrued Interest on Pledged Assets	21,764
Repurchase Agreements and Other Advances | Agency MBS
Collateral Positions
Fair Value	2,531,106
Amortized Cost	2,559,275
Accrued Interest on Pledged Assets	6,693
Repurchase Agreements and Other Advances | Legacy Non-Agency MBS
Collateral Positions
Fair Value	1,897,016
Amortized Cost	1,456,018
Accrued Interest on Pledged Assets	7,268
Non-Agency MBS pledged as collateral in connection with contemporaneous repurchase and reverse repurchase agreements entered into with a single counterparty	329,800
Repurchase Agreements and Other Advances | RPL/NPL MBS
Collateral Positions
Fair Value	733,409
Amortized Cost	731,571
Accrued Interest on Pledged Assets	576
Repurchase Agreements and Other Advances | U.S. Treasury securities
Collateral Positions
Fair Value	220,814
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Repurchase Agreements and Other Advances | CRT securities
Collateral Positions
Fair Value	590,551
Amortized Cost	538,092
Accrued Interest on Pledged Assets	488
Repurchase Agreements and Other Advances | MSR related assets
Collateral Positions
Fair Value	432,468
Amortized Cost	431,429
Accrued Interest on Pledged Assets	1,084
Repurchase Agreements and Other Advances | Residential whole loans
Collateral Positions
Fair Value	1,706,627
Amortized Cost	1,676,218
Accrued Interest on Pledged Assets	5,655
Repurchase Agreements and Other Advances | Cash
Collateral Positions
Fair Value	7,100
Amortized Cost	7,100
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments
Collateral Positions
Fair Value	16,244
Amortized Cost	17,097
Accrued Interest on Pledged Assets	34
Derivative Hedging Instruments | Agency MBS
Collateral Positions
Fair Value	16,244
Amortized Cost	17,097
Accrued Interest on Pledged Assets	34
Derivative Hedging Instruments | Legacy Non-Agency MBS
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | RPL/NPL MBS
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | U.S. Treasury securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | CRT securities
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | MSR related assets
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Residential whole loans
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Derivative Hedging Instruments | Cash
Collateral Positions
Fair Value	0
Amortized Cost	0
Accrued Interest on Pledged Assets	0
Carrying Value | Repurchase Agreements and Other Advances | Residential whole loans
Collateral Positions
Fair Value	468,600
Amortized Cost	438,200
Total Fair Value of Assets Pledged and Accrued Interest	$ 1,200,000