Residential Whole Loans (LTV on Loans) (Details) - 90 Days or More Past Due $ in Thousands	Dec. 31, 2020 USD ($)
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 119,621
UPB	$ 145,028
LTV	0.867
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 148,387
UPB	$ 144,681
LTV	0.659
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 136,347
UPB	$ 136,347
LTV	0.658
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 20,388
UPB	$ 20,233
LTV	0.727
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 8,031
UPB	$ 8,823
LTV	0.551
Residential Loans At Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 571,729
UPB	$ 625,621
LTV	0.868