Residential Whole Loans (Residential Whole Loans, at Carrying Value) (Details) $ in Thousands	Dec. 31, 2020 USD ($) loan	Jan. 01, 2020 USD ($)	Dec. 31, 2019 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 3,521,624		$ 5,370,653
Purchased credit impaired loans	673,708		698,717
Total Residential whole loans, at carrying value	4,195,332		6,069,370	[1]
Allowance for credit losses on residential whole loans held at carrying value	(86,833)		(3,025)
Total Residential whole loans at carrying value, net	$ 4,108,499		$ 6,066,345
Number of loans | loan	13,112		17,082
Cumulative Effect, Period of Adoption, Adjustment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans		$ 62,600
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 2,357,185		$ 3,707,245
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	581,801		1,026,097
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	446,374		460,742
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 136,264		$ 176,569

[1] For the year ended December 31, 2020, includes approximately $2.7 million (360,534 shares) surrendered for tax purposes related to equity-based compensation awards. For the year ended December 31, 2019, includes approximately $4.1 million (562,815 shares) surrendered for tax purposes related to equity-based compensation awards. For the year ended December 31, 2018, includes approximately $3.4 million (464,429 shares) surrendered for tax purposes related to equity-based compensation awards.