Repurchase Agreements and Other Advances (Details 2) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Information about MBS [Line Items]
Less unamortized debt issuance costs	$ 441	$ 210
Securities Sold under Agreements to Repurchase, Amounts Not Included in Offsetting Disclosure	0
Repurchase Agreement Borrowings
Information about MBS [Line Items]
Balance of repurchase agreements	$ 8,137,543	$ 8,472,478
Weighted Average Interest Rate	1.91%	1.48%
Less unamortized debt issuance costs	$ 441	$ 210
Total repurchase agreements less debt issuance costs	8,137,102	8,472,268
Repurchase Agreement Borrowings | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	$ 5,656,600
Weighted Average Interest Rate	1.56%
Repurchase Agreement Borrowings | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 658,265
Weighted Average Interest Rate	2.41%
Repurchase Agreement Borrowings | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 1,792,070
Weighted Average Interest Rate	2.80%
Repurchase Agreement Borrowings | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 30,608
Weighted Average Interest Rate	3.36%
Repurchase Agreement Borrowings | Agency MBS
Information about MBS [Line Items]
Balance of repurchase agreements	$ 3,093,737	3,095,020
Repurchase Agreement Borrowings | Agency MBS | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	3,093,737
Repurchase Agreement Borrowings | Agency MBS | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS
Information about MBS [Line Items]
Balance of repurchase agreements	1,573,545	1,690,937
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	926,866
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	336,316
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	310,363
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | 3 Year Step-up securities
Information about MBS [Line Items]
Balance of repurchase agreements	1,875,157	2,035,531
Repurchase Agreement Borrowings | 3 Year Step-up securities | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | 3 Year Step-up securities | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	894,904
Repurchase Agreement Borrowings | 3 Year Step-up securities | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	304,141
Repurchase Agreement Borrowings | 3 Year Step-up securities | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	676,112
Repurchase Agreement Borrowings | 3 Year Step-up securities | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries
Information about MBS [Line Items]
Balance of repurchase agreements	366,475	504,572
Repurchase Agreement Borrowings | U.S. Treasuries | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	366,475
Repurchase Agreement Borrowings | U.S. Treasuries | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities
Information about MBS [Line Items]
Balance of repurchase agreements	345,482	271,205
Repurchase Agreement Borrowings | CRT securities | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	327,674
Repurchase Agreement Borrowings | CRT securities | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	17,808
Repurchase Agreement Borrowings | CRT securities | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans
Information about MBS [Line Items]
Balance of repurchase agreements	836,203	832,060
Repurchase Agreement Borrowings | Residential whole loans | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	805,595
Repurchase Agreement Borrowings | Residential whole loans | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	30,608
Repurchase Agreement Borrowings | Other investments
Information about MBS [Line Items]
Balance of repurchase agreements	46,944	43,333
Repurchase Agreement Borrowings | Other investments | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Other investments | Within 30 Days
Information about MBS [Line Items]
Balance of repurchase agreements	46,944
Repurchase Agreement Borrowings | Other investments | Over 30 Days to 3 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Other investments | Over 3 Months to 12 Months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Other investments | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	$ 7,502,995	$ 7,284,062
Weighted Average Interest Rate	1.86%	1.77%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 634,548	$ 1,188,416
Weighted Average Interest Rate	2.41%	1.91%