Repurchase Agreements (Borrowings Under Repurchase Agreement) (Details) - USD ($) $ in Thousands	Mar. 31, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 8,509,713	$ 7,879,087
Repurchase Agreement Borrowings
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 8,509,755	$ 7,879,114
Weighted Average Interest Rate	3.52%	3.42%
Less debt issuance costs	$ 42	$ 27
Total repurchase agreements less debt issuance costs	8,509,713	7,879,087
Repurchase Agreement Borrowings | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 7,211,430
Weighted Average Interest Rate	3.51%
Repurchase Agreement Borrowings | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 669,143
Weighted Average Interest Rate	2.97%
Repurchase Agreement Borrowings | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 629,182
Weighted Average Interest Rate	4.18%
Repurchase Agreement Borrowings | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 7,265,257	$ 6,747,166
Weighted Average Interest Rate	3.51%	3.35%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 669,143	$ 368,857
Weighted Average Interest Rate	2.97%	3.10%
Repurchase Agreement Borrowings | Over 3 months to 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 575,355	$ 763,091
Weighted Average Interest Rate	4.18%	4.18%
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 2,353,173	$ 2,384,357
Agency MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Agency MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,929,331
Agency MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	423,842
Agency MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Agency MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Legacy Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,359,699	1,447,585
Legacy Non-Agency MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Legacy Non-Agency MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,316,983
Legacy Non-Agency MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	42,716
Legacy Non-Agency MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Legacy Non-Agency MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	1,009,331	1,084,532
RPL/NPL MBS | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	964,103
RPL/NPL MBS | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	45,228
RPL/NPL MBS | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
RPL/NPL MBS | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	338,827	391,586
CRT securities | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	314,095
CRT securities | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	24,732
CRT securities | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
CRT securities | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Residential Whole Loans, at Fair Value
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	2,746,804	2,020,508
Residential Whole Loans, at Fair Value | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Residential Whole Loans, at Fair Value | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	2,219,985
Residential Whole Loans, at Fair Value | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Residential Whole Loans, at Fair Value | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	526,819
Residential Whole Loans, at Fair Value | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
MSR-related assets
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	647,535	474,127
MSR-related assets | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
MSR-related assets | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	461,083
MSR-related assets | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	132,625
MSR-related assets | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	53,827
MSR-related assets | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	54,386	$ 76,419
Other | Overnight
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other | Within 30 Days
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	5,850
Other | Over 30 Days to 3 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	0
Other | Over 3 Months to 12 Months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	48,536
Other | Over 12 months
Debt Securities, Available-for-sale [Line Items]
Balance of repurchase agreements	$ 0