Residential Whole Loans - LTV on Loans (Details) - 90 or more $ in Thousands	Jun. 30, 2023 USD ($)	Dec. 31, 2022 USD ($)
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 202,513	$ 208,659
UPB	209,507	211,164
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	540,194	622,641
UPB	581,091	665,511
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	266,954	330,810
UPB	$ 283,045	$ 350,589
LTV (as a percent)	0.763	0.756
Non-QM loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 93,842	$ 93,508
UPB	$ 95,784	$ 93,972
LTV (as a percent)	0.663	0.667
Transitional loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 68,110	$ 75,449
UPB	$ 71,788	$ 76,415
LTV (as a percent)	0.672	0.681
Single-family rental loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 36,540	$ 34,653
UPB	$ 37,763	$ 35,441
LTV (as a percent)	0.719	0.721
Seasoned performing loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 3,185	$ 5,049
UPB	$ 3,179	$ 5,336
LTV (as a percent)	0.375	0.417
Agency eligible investor loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 836	$ 0
UPB	$ 993	$ 0
LTV (as a percent)	0.730	0
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 70,727	$ 83,172
UPB	$ 88,539	$ 103,758
LTV (as a percent)	0.726	0.725