Fair Value of Financial Instruments (Fair Value Hierarchy) (Details) - USD ($) $ in Thousands	Dec. 31, 2018	Dec. 31, 2017
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	$ 6,509,333	$ 7,023,050
Residential whole loans, at fair value	1,665,978	1,325,115
Securities obtained and pledged as collateral	0	504,062
Total assets carried at fair value	8,713,810	9,234,031
Obligation to return securities held as collateral, at fair value	0	504,062
Total liabilities carried at fair value	0	504,062
Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	2,698,213	2,824,681
Non-Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	3,318,299	3,533,966
CRT securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	492,821	664,403
Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	1,665,978	1,325,115
Term notes backed by MSR-related collateral
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Term notes backed by MSR-related collateral	538,499	381,804
Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Securities obtained and pledged as collateral	0	504,062
Total assets carried at fair value	0	504,062
Obligation to return securities held as collateral, at fair value	0	504,062
Total liabilities carried at fair value	0	504,062
Level 1 | Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 1 | Non-Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 1 | CRT securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 1 | Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	0	0
Level 1 | Term notes backed by MSR-related collateral
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Term notes backed by MSR-related collateral	0	0
Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Securities obtained and pledged as collateral	0	0
Total assets carried at fair value	6,509,333	7,023,050
Obligation to return securities held as collateral, at fair value	0	0
Total liabilities carried at fair value	0	0
Level 2 | Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	2,698,213	2,824,681
Level 2 | Non-Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	3,318,299	3,533,966
Level 2 | CRT securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	492,821	664,403
Level 2 | Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	0	0
Level 2 | Term notes backed by MSR-related collateral
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Term notes backed by MSR-related collateral	0	0
Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Securities obtained and pledged as collateral	0	0
Total assets carried at fair value	2,204,477	1,706,919
Obligation to return securities held as collateral, at fair value	0	0
Total liabilities carried at fair value	0	0
Level 3 | Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 3 | Non-Agency MBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 3 | CRT securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Available-for-sale Securities	0	0
Level 3 | Residential whole loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	1,665,978	1,325,115
Level 3 | Term notes backed by MSR-related collateral
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Term notes backed by MSR-related collateral	$ 538,499	$ 381,804