Fair Value of Financial Instruments (Fair Value Hierarchy) (Details) - USD ($) $ in Thousands		Sep. 30, 2020	Dec. 31, 2019
Assets:
Residential whole loans, at fair value	[1]	$ 1,229,664	$ 1,381,583
Fair  Value		152,765	3,983,519
Total assets carried at fair value		1,616,520	6,522,565
Liabilities:
Securitized debt		388,790
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,229,664	1,381,583
Liabilities:
Total liabilities carried at fair value		4,080,461
Non-Agency MBS
Assets:
Fair  Value		56,430	2,063,529
Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Assets:
Fair  Value			1,664,582
CRT securities
Assets:
Fair  Value		96,335	255,408
Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		234,091	1,157,463
Financing agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,727,407
Financing agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,490,271
Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		473,993
Level 1
Assets:
Total assets carried at fair value		0	0
Liabilities:
Securitized debt		0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		0
Level 1 | Non-Agency MBS
Assets:
Fair  Value		0	0
Level 1 | Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Assets:
Fair  Value			0
Level 1 | CRT securities
Assets:
Fair  Value		0	0
Level 1 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	0
Level 1 | Financing agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 1 | Financing agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 1 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		0
Level 2
Assets:
Total assets carried at fair value		386,856	5,140,982
Liabilities:
Securitized debt		388,790
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		1,121,320
Level 2 | Non-Agency MBS
Assets:
Fair  Value		56,430	2,063,529
Level 2 | Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Assets:
Fair  Value			1,664,582
Level 2 | CRT securities
Assets:
Fair  Value		96,335	255,408
Level 2 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		234,091	1,157,463
Level 2 | Financing agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		0
Level 2 | Financing agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		258,537
Level 2 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		473,993
Level 3
Assets:
Total assets carried at fair value		1,229,664	1,381,583
Liabilities:
Securitized debt		0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,229,664	1,381,583
Liabilities:
Total liabilities carried at fair value		2,959,141
Level 3 | Non-Agency MBS
Assets:
Fair  Value		0	0
Level 3 | Mortgage-backed Securities, Issued by US Government Sponsored Enterprises [Member]
Assets:
Fair  Value			0
Level 3 | CRT securities
Assets:
Fair  Value		0	0
Level 3 | Term notes backed by MSR-related collateral
Assets:
Term notes backed by MSR related collateral		0	$ 0
Level 3 | Financing agreements with non-mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,727,407
Level 3 | Financing agreements with mark-to-market collateral provisions
Liabilities:
Liabilities carried at fair value		1,231,734
Level 3 | Senior secured credit agreement
Liabilities:
Liabilities carried at fair value		$ 0

[1] Includes approximately $568.6 million and $186.4 million of Residential whole loans, at carrying value and $521.2 million and $567.4 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at September 30, 2020 and December 31, 2019, respectively. Such assets can be used only to settle the obligations of each respective VIE.