Residential Whole Loans - LTV on Loans (Details) - 90 or more $ in Thousands	Jun. 30, 2022 USD ($)	Dec. 31, 2021 USD ($)
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 196,618	$ 230,903
UPB	197,249	229,520
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	665,457	781,245
UPB	690,942	800,145
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	385,233	454,443
UPB	$ 389,901	$ 453,146
LTV (as a percent)	0.797	0.802
Non-QM loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 78,037	$ 96,473
UPB	$ 77,523	$ 94,755
LTV (as a percent)	0.666	0.646
Rehabilitation loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 81,432	$ 103,166
UPB	$ 81,992	$ 103,034
LTV (as a percent)	0.708	0.676
Single-family rental loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 29,577	$ 23,524
UPB	$ 29,493	$ 23,487
LTV (as a percent)	0.752	0.734
Seasoned performing loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 6,854	$ 7,740
UPB	$ 7,427	$ 8,244
LTV (as a percent)	0.454	0.456
Agency eligible investor loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 718	$ 0
UPB	$ 814	$ 0
LTV (as a percent)	0.592	0
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 83,606	$ 95,899
UPB	$ 103,792	$ 117,479
LTV (as a percent)	0.771	0.791