Fair Value of Financial Instruments (Fair Value Hierarchy) (Details) - USD ($) $ in Thousands		Mar. 31, 2021	Dec. 31, 2020
Assets:
Residential whole loans, at fair value	[1]	$ 1,320,199	$ 1,216,902
Total assets carried at fair value		1,670,314	1,616,901
Liabilities:
Securitized debt		753,008	869,482
Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,320,199	1,216,902
Liabilities:
Total liabilities carried at fair value		2,974,578	3,366,772
Securities, at fair value
Assets:
Securities, at fair value		350,115	399,999
Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		1,041,283	1,159,213
Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		1,180,287	1,338,077
Level 1
Assets:
Total assets carried at fair value		0	0
Liabilities:
Securitized debt		0	0
Level 1 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		0	0
Level 1 | Securities, at fair value
Assets:
Securities, at fair value		0	0
Level 1 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		0	0
Level 1 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		0	0
Level 2
Assets:
Total assets carried at fair value		350,115	399,999
Liabilities:
Securitized debt		753,008	869,482
Level 2 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		0	0
Liabilities:
Total liabilities carried at fair value		953,754	1,083,397
Level 2 | Securities, at fair value
Assets:
Securities, at fair value		350,115	399,999
Level 2 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		0	0
Level 2 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		200,746	213,915
Level 3
Assets:
Total assets carried at fair value		1,320,199	1,216,902
Liabilities:
Securitized debt		0	0
Level 3 | Residential whole loans, at fair value
Assets:
Residential whole loans, at fair value		1,320,199	1,216,902
Liabilities:
Total liabilities carried at fair value		2,020,824	2,283,375
Level 3 | Securities, at fair value
Assets:
Securities, at fair value		0	0
Level 3 | Agreements with Non-mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		1,041,283	1,159,213
Level 3 | Agreements with Mark-to-market Collateral Provisions
Liabilities:
Liabilities carried at fair value		$ 979,541	$ 1,124,162

[1] Includes approximately $1.5 billion and $1.4 billion of Residential whole loans, at carrying value and $311.6 million and $382.3 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at March 31, 2021 and December 31, 2020, respectively. Such assets can be used only to settle the obligations of each respective VIE.