Residential Whole Loans (Residential Whole Loans, at Carrying Value) (Details) $ in Thousands		Mar. 31, 2021 USD ($) loan	Dec. 31, 2020 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans		$ 3,287,689	$ 3,521,624
Purchased Credit Deteriorated Loans		644,611	673,708
Total Residential whole loans, at carrying value	[1]	3,932,300	4,195,332
Allowance for credit losses on residential whole loans held at carrying value		(63,244)	(86,833)
Total Residential whole loans at carrying value, net		$ 3,869,056	$ 4,108,499
Number of loans | loan		12,575	13,112
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans		$ 2,243,444	$ 2,357,185
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans		464,385	581,801
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans		451,791	446,374
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans		$ 128,069	$ 136,264

[1] Includes approximately $1.5 billion and $1.4 billion of Residential whole loans, at carrying value and $311.6 million and $382.3 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at March 31, 2021 and December 31, 2020, respectively. Such assets can be used only to settle the obligations of each respective VIE.