Repurchase Agreements and Other Advances (Details 2) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Schedule of Available-for-sale Securities [Line Items]
Less unamortized debt issuance costs	$ 259	$ 210
Amounts related to repurchase agreements not included in offsetting disclosure in Note 8	0
Repurchase Agreement Borrowings
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 6,871,702	$ 8,472,478
Weighted Average Interest Rate	2.17%	1.79%
Less unamortized debt issuance costs	$ 259	$ 210
Total repurchase agreements less debt issuance costs	6,871,443	8,472,268
Repurchase Agreement Borrowings | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 4,848,957
Weighted Average Interest Rate	1.83%
Repurchase Agreement Borrowings | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 880,243
Weighted Average Interest Rate	2.56%
Repurchase Agreement Borrowings | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 1,080,906
Weighted Average Interest Rate	3.31%
Repurchase Agreement Borrowings | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 61,596
Weighted Average Interest Rate	3.68%
Repurchase Agreement Borrowings | Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 2,671,245	3,095,020
Repurchase Agreement Borrowings | Agency MBS | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	2,590,020
Repurchase Agreement Borrowings | Agency MBS | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	81,225
Repurchase Agreement Borrowings | Agency MBS | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	1,361,866	1,690,937
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	746,798
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	478,331
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	136,737
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | RPL/NPL MBS
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	798,508	1,943,572
Repurchase Agreement Borrowings | RPL/NPL MBS | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | RPL/NPL MBS | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	359,471
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	316,638
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	122,399
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	474,726	504,572
Repurchase Agreement Borrowings | U.S. Treasuries | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	474,726
Repurchase Agreement Borrowings | U.S. Treasuries | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	413,172	271,205
Repurchase Agreement Borrowings | CRT securities | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	409,123
Repurchase Agreement Borrowings | CRT securities | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	4,049
Repurchase Agreement Borrowings | CRT securities | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | MSR related assets
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	268,819	135,112
Repurchase Agreement Borrowings | MSR related assets | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | MSR related assets | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	268,819
Repurchase Agreement Borrowings | MSR related assets | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | MSR related assets | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | MSR related assets | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	883,366	832,060
Repurchase Agreement Borrowings | Residential whole loans | Overnight
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Within 30 Days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Over 30 Days to 3 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential whole loans | Over 3 Months to 12 Months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	821,770
Repurchase Agreement Borrowings | Residential whole loans | Over 12 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	61,596
Repurchase Agreement Borrowings | Within 30 days
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 6,378,566	$ 7,284,062
Weighted Average Interest Rate	2.15%	1.77%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Schedule of Available-for-sale Securities [Line Items]
Balance of repurchase agreements	$ 493,136	$ 1,188,416
Weighted Average Interest Rate	2.42%	1.91%