Fair Value of Financial Instruments (Fair Value Hierarchy) (Details) - USD ($)		Mar. 31, 2019	Dec. 31, 2018
Assets:
Available-for-sale securities		$ 6,069,571,000	$ 6,509,333,000
Residential whole loans, at fair value	[1]	1,512,337,000	1,665,978,000
Total assets carried at fair value		8,335,502,000	8,713,810,000
Agency MBS
Assets:
Available-for-sale securities		2,546,597,000	2,698,213,000
Non-Agency MBS
Assets:
Available-for-sale securities		3,099,272,000	3,318,299,000
CRT securities
Assets:
Available-for-sale securities		423,702,000	492,821,000
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,512,337,000	1,665,978,000
Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		753,594,000	538,499,000
Level 1
Assets:
Total assets carried at fair value		0	0
Level 1 | Agency MBS
Assets:
Available-for-sale securities		0	0
Level 1 | Non-Agency MBS
Assets:
Available-for-sale securities		0	0
Level 1 | CRT securities
Assets:
Available-for-sale securities		0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Level 1 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	0
Level 2
Components of financial instruments carried at fair value
Credit valuation adjustment to derivative liabilities		0
Credit valuation adjustment to derivative assets		0
Assets:
Total assets carried at fair value		6,069,571,000	6,509,333,000
Level 2 | Agency MBS
Assets:
Available-for-sale securities		2,546,597,000	2,698,213,000
Level 2 | Non-Agency MBS
Assets:
Available-for-sale securities		3,099,272,000	3,318,299,000
Level 2 | CRT securities
Assets:
Available-for-sale securities		423,702,000	492,821,000
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Level 2 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	0
Level 3
Assets:
Total assets carried at fair value		2,265,931,000	2,204,477,000
Level 3 | Agency MBS
Assets:
Available-for-sale securities		0	0
Level 3 | Non-Agency MBS
Assets:
Available-for-sale securities		0	0
Level 3 | CRT securities
Assets:
Available-for-sale securities		0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,512,337,000	1,665,978,000
Level 3 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		$ 753,594,000	$ 538,499,000

[1] Includes approximately $202.7 million and $209.4 million of Residential whole loans, at carrying value and $647.0 million and $694.7 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at March 31, 2019 and December 31, 2018, respectively. Such assets can be used only to settle the obligations of each respective VIE.