Residential Whole Loans - LTV on Loans (Details) - Financial Asset, Equal to or Greater than 60 Days Past Due $ in Thousands	Mar. 31, 2024 USD ($)	Dec. 31, 2023 USD ($)
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 333,861	$ 284,391
UPB	359,614	307,732
Carrying Value | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	126,310	102,252
UPB	$ 129,974	$ 104,454
LTV (as a percent)	0.652	0.639
Carrying Value | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 142,423	$ 113,772
UPB	$ 150,938	$ 119,936
LTV (as a percent)	0.659	0.651
Carrying Value | Single-family Rental Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 63,173	$ 65,659
UPB	$ 76,621	$ 80,570
LTV (as a percent)	1.110	1.091
Carrying Value | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 1,373	$ 2,520
UPB	$ 1,371	$ 2,544
LTV (as a percent)	0.246	0.336
Carrying Value | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 582	$ 188
UPB	$ 710	$ 228
LTV (as a percent)	0.717	0.734
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 609,730	$ 572,799
UPB	664,367	621,641
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	212,437	222,319
UPB	$ 225,305	$ 231,782
LTV (as a percent)	0.696	0.707
Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 63,432	$ 66,089
UPB	$ 79,448	$ 82,127
LTV (as a percent)	0.643	0.643