Residential Whole Loans (LTV on Loans) (Details) $ in Thousands	Jun. 30, 2021 USD ($)	Dec. 31, 2020 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unsettled residential whole loans	$ 131,300
90 Days or More Delinquent | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	386,198	$ 432,774
UPB	406,728	455,112
90 Days or More Delinquent | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	513,585	571,729
UPB	530,575	625,621
90 Days or More Delinquent | Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	510,611	571,729
UPB	$ 527,572	$ 625,621
LTV (as a percent)	0.822	0.868
90 Days or More Delinquent | Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 2,974	$ 0
UPB	$ 3,003	$ 0
LTV (as a percent)	0.653	0
90 Days or More Delinquent | Purchased credit deteriorated loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 105,487	$ 119,621
UPB	$ 128,147	$ 145,028
LTV (as a percent)	0.846	0.867
90 Days or More Delinquent | Non-QM loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 140,051	$ 148,387
UPB	$ 137,252	$ 144,681
LTV (as a percent)	0.658	0.659
90 Days or More Delinquent | Rehabilitation loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 108,942	$ 136,347
UPB	$ 108,942	$ 136,347
LTV (as a percent)	0.679	0.658
90 Days or More Delinquent | Single-family rental loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 22,595	$ 20,388
UPB	$ 22,595	$ 20,233
LTV (as a percent)	0.737	0.727
90 Days or More Delinquent | Seasoned performing loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 9,123	$ 8,031
UPB	$ 9,792	$ 8,823
LTV (as a percent)	0.501	0.551