Use of Special Purpose Entities and Variable Interest Entities (Narrative) (Details) - USD ($) $ in Thousands		3 Months Ended	9 Months Ended
		Sep. 30, 2020	Sep. 30, 2020	Sep. 30, 2019	Dec. 31, 2019
Variable Interest Entity [Line Items]
Residential whole loans, carrying value	[1]	$ 4,493,805	$ 4,493,805		$ 6,069,370
Residential whole loans, at fair value	[1]	1,229,664	1,229,664		1,381,583
Residential whole loans, net		4,387,559	4,387,559		6,066,345
Total residential whole loans		5,600,000	5,600,000		7,400,000
Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Proceeds from issuance of debt			1,193,969	$ 802,815
Residential whole loans, carrying value		568,600	568,600		186,400
Residential whole loans, at fair value		521,200	521,200		567,400
Securitized debt		837,700	837,700		571,000
Asset-backed Securities, Securitized Loans and Receivables | Other Assets
Variable Interest Entity [Line Items]
Real estate owned at fair value		80,300	80,300		$ 137,800
Senior Bonds
Variable Interest Entity [Line Items]
Proceeds from issuance of debt		372,800	399,800
Proceed from debt net of offering expenses and underwriting discount		$ 372,800	$ 391,200

[1] Includes approximately $568.6 million and $186.4 million of Residential whole loans, at carrying value and $521.2 million and $567.4 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at September 30, 2020 and December 31, 2019, respectively. Such assets can be used only to settle the obligations of each respective VIE.