Fair Value of Financial Instruments (Fair Value Hierarchy) (Details) - USD ($) $ in Thousands		Jun. 30, 2020	Dec. 31, 2019
Assets:
Residential whole loans, at fair value	[1]	$ 1,200,981	$ 1,381,583
Available-for-sale securities		148,343	3,983,519
Total assets carried at fair value		1,574,106	6,522,565
Liabilities:
Other		20,520
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,200,981	1,381,583
Liabilities:
Total liabilities carried at fair value		4,194,324
Non-Agency MBS
Assets:
Available-for-sale securities		54,237	2,063,529
Agency MBS
Assets:
Available-for-sale securities			1,664,582
CRT securities
Assets:
Available-for-sale securities		94,106	255,408
Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		224,782	1,157,463
Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		2,036,597
Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,656,248
Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		480,959
Level 1
Assets:
Total assets carried at fair value		0	0
Liabilities:
Other		0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		0
Level 1 | Non-Agency MBS
Assets:
Available-for-sale securities		0	0
Level 1 | Agency MBS
Assets:
Available-for-sale securities			0
Level 1 | CRT securities
Assets:
Available-for-sale securities		0	0
Level 1 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	0
Level 1 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 1 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 1 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		0
Level 2
Assets:
Total assets carried at fair value		373,125	5,140,982
Liabilities:
Other		20,520
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		4,194,324
Level 2 | Non-Agency MBS
Assets:
Available-for-sale securities		54,237	2,063,529
Level 2 | Agency MBS
Assets:
Available-for-sale securities			1,664,582
Level 2 | CRT securities
Assets:
Available-for-sale securities		94,106	255,408
Level 2 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		224,782	1,157,463
Level 2 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		2,036,597
Level 2 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,656,248
Level 2 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		480,959
Level 3
Assets:
Total assets carried at fair value		1,200,981	1,381,583
Liabilities:
Other		0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,200,981	1,381,583
Liabilities:
Total liabilities carried at fair value		0
Level 3 | Non-Agency MBS
Assets:
Available-for-sale securities		0	0
Level 3 | Agency MBS
Assets:
Available-for-sale securities			0
Level 3 | CRT securities
Assets:
Available-for-sale securities		0	0
Level 3 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	$ 0
Level 3 | Agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 3 | Agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 3 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		$ 0

[1] Includes approximately $181.4 million and $186.4 million of Residential whole loans, at carrying value and $506.7 million and $567.4 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at June 30, 2020 and December 31, 2019, respectively. Such assets can be used only to settle the obligations of each respective VIE.