Residential Whole Loans - LTV on Loans (Details) - 90 or more $ in Thousands	Sep. 30, 2022 USD ($)	Dec. 31, 2021 USD ($)
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 173,350	$ 230,903
UPB	174,657	229,520
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	581,959	781,245
UPB	614,194	800,145
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	331,847	454,443
UPB	$ 343,229	$ 453,146
LTV (as a percent)	0.785	0.802
Non-QM loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 64,036	$ 96,473
UPB	$ 63,831	$ 94,755
LTV (as a percent)	0.677	0.646
Residential transition loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 76,514	$ 103,166
UPB	$ 77,510	$ 103,034
LTV (as a percent)	0.699	0.676
Single-family rental loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 25,901	$ 23,524
UPB	$ 25,753	$ 23,487
LTV (as a percent)	0.749	0.734
Seasoned performing loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 5,850	$ 7,740
UPB	$ 6,345	$ 8,244
LTV (as a percent)	0.454	0.456
Agency eligible investor loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 1,049	$ 0
UPB	$ 1,218	$ 0
LTV (as a percent)	0.617	0
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 76,762	$ 95,899
UPB	$ 96,308	$ 117,479
LTV (as a percent)	0.758	0.791