Residential Whole Loans - LTV on Loans (Details) - 90 or more $ in Thousands	Dec. 31, 2022 USD ($)	Dec. 31, 2021 USD ($)
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 532,770	$ 781,245
Held at Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	296,697	454,443
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	532,770	781,245
UPB	565,284	800,145
Held at Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	296,697	454,443
UPB	$ 310,035	$ 453,146
LTV (as a percent)	0.769	0.802
Non-QM loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 61,812	$ 96,473
UPB	$ 61,719	$ 94,755
LTV (as a percent)	0.679	0.646
Non-QM loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 61,812	$ 96,473
Transitional loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	73,266	103,166
UPB	$ 74,180	$ 103,034
LTV (as a percent)	0.681	0.676
Transitional loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 73,266	$ 103,166
Single-family rental loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	27,466	23,524
UPB	$ 27,463	$ 23,487
LTV (as a percent)	0.729	0.734
Single-family rental loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 27,466	$ 23,524
Seasoned performing loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	4,127	7,740
UPB	$ 4,399	$ 8,244
LTV (as a percent)	0.422	0.456
Seasoned performing loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 4,127	$ 7,740
Agency eligible investor loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	0	0
UPB	$ 0	$ 0
LTV (as a percent)	0	0
Agency eligible investor loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 0	$ 0
Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	166,671	230,903
UPB	167,761	229,520
Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	166,671	230,903
Purchased Credit Deteriorated Loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	69,402	95,899
Purchased Credit Deteriorated Loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	69,402	95,899
UPB	$ 87,488	$ 117,479
LTV (as a percent)	0.748	0.791