Use of Special Purpose Entities and Variable Interest Entities (Narrative) (Details) - USD ($) $ in Thousands		3 Months Ended	6 Months Ended
		Jun. 30, 2020	Jun. 30, 2020	Jun. 30, 2019	Dec. 31, 2019
Variable Interest Entity [Line Items]
Residential whole loans, carrying value	[1]	$ 4,813,119	$ 4,813,119		$ 6,069,370
Residential whole loans, at fair value	[1]	1,200,981	1,200,981		1,381,583
Residential whole loans, net		4,676,530	4,676,530		6,066,345
Total residential whole loans		5,900,000	5,900,000		7,400,000
Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Debt instrument, face amount		27,000	27,000
Proceeds from issuance of debt		18,400	821,175	$ 802,815
Residential whole loans, carrying value		181,400	181,400		186,400
Residential whole loans, at fair value		506,700	506,700		567,400
Securitized debt		516,100	516,100		571,000
Asset-backed Securities, Securitized Loans and Receivables | Other Assets
Variable Interest Entity [Line Items]
Real estate owned at fair value		98,700	98,700		137,800
Senior Bonds | Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Debt instrument, face amount		$ 829,817	$ 829,817		$ 802,817

[1] Includes approximately $181.4 million and $186.4 million of Residential whole loans, at carrying value and $506.7 million and $567.4 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at June 30, 2020 and December 31, 2019, respectively. Such assets can be used only to settle the obligations of each respective VIE.