Residential Whole Loans - LTV on Loans (Details) - 90 or more $ in Thousands	Mar. 31, 2023 USD ($)	Dec. 31, 2022 USD ($)
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 213,290	$ 208,659
UPB	217,921	211,164
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	600,551	622,641
UPB	646,883	665,511
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	303,400	330,810
UPB	$ 324,696	$ 350,589
LTV (as a percent)	0.763	0.756
Non-QM loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 99,532	$ 93,508
UPB	$ 100,167	$ 93,972
LTV (as a percent)	0.659	0.667
Transitional loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 71,299	$ 75,449
UPB	$ 73,438	$ 76,415
LTV (as a percent)	0.664	0.681
Single-family rental loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 37,839	$ 34,653
UPB	$ 39,340	$ 35,441
LTV (as a percent)	0.720	0.721
Seasoned performing loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 4,213	$ 5,049
UPB	$ 4,486	$ 5,336
LTV (as a percent)	0.484	0.417
Agency eligible investor loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 407	$ 0
UPB	$ 490	$ 0
LTV (as a percent)	0.646	0
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 83,861	$ 83,172
UPB	$ 104,266	$ 103,758
LTV (as a percent)	0.724	0.725