Repurchase Agreements (Borrowings Under Repurchase Agreement) (Details) - USD ($) $ in Thousands	6 Months Ended
	Jun. 30, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 8,630,642	$ 7,879,087
Repurchase agreement financings are subject to termination	$ 1,400,000
Repurchase agreement financings are subject to termination, period	6 months
Repurchase Agreement Borrowings
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 8,630,707	$ 7,879,114
Weighted Average Interest Rate	3.47%	3.42%
Less debt issuance costs	$ 65	$ 27
Total repurchase agreements less debt issuance costs	8,630,642	7,879,087
Repurchase Agreement Borrowings | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 5,039,389
Weighted Average Interest Rate	3.09%
Repurchase Agreement Borrowings | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 1,401,643
Weighted Average Interest Rate	4.04%
Repurchase Agreement Borrowings | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 1,484,173
Weighted Average Interest Rate	3.92%
Repurchase Agreement Borrowings | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 705,502
Weighted Average Interest Rate	4.14%
Repurchase Agreement Borrowings | Within 30 days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 7,691,365	$ 6,747,166
Weighted Average Interest Rate	3.43%	3.35%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 255,437	$ 368,857
Weighted Average Interest Rate	3.28%	3.10%
Repurchase Agreement Borrowings | Over 3 months to 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 683,905	$ 763,091
Weighted Average Interest Rate	4.04%	4.18%
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 2,065,386	$ 2,384,357
Agency MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Agency MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,955,072
Agency MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	110,314
Agency MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Agency MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,276,010	1,447,585
Legacy Non-Agency MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Legacy Non-Agency MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,006,126
Legacy Non-Agency MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	42,750
Legacy Non-Agency MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	227,134
Legacy Non-Agency MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	808,969	1,084,532
RPL/NPL MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	808,969
RPL/NPL MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	324,560	391,586
CRT securities | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	324,560
CRT securities | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Residential Whole Loans, at Fair Value
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	3,148,919	2,020,508
Residential Whole Loans, at Fair Value | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Residential Whole Loans, at Fair Value | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	194,068
Residential Whole Loans, at Fair Value | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,248,579
Residential Whole Loans, at Fair Value | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,000,770
Residential Whole Loans, at Fair Value | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	705,502
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	920,733	474,127
MSR-related assets | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
MSR-related assets | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	744,743
MSR-related assets | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
MSR-related assets | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	175,990
MSR-related assets | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	86,130	$ 76,419
Other | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	5,851
Other | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	80,279
Other | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 0