Residential Whole Loans - Residential Whole Loans, at Carrying Value Additional Information (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2023	Dec. 31, 2022
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Loans	$ 1,670,788
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	912,826
Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	42,427
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	191,780
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 75,389
Multi-Family Transitional Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	64.00%	64.00%
Loans	$ 821,500	$ 632,300
Financing Receivable, Weighted Average Term To Maturity	15 months	18 months
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	$ 6,971,917	$ 6,273,743
Unpaid Principal Balance (“UPB”)	$ 7,434,036	$ 6,741,786
Weighted Average Coupon	6.26%	5.78%
Weighted Average Term to Maturity (Months)	260 months	271 months
60+ Delinquency %	2.80%	3.10%
Settled Whole Loans | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	$ 3,602,412	$ 3,352,471
Unpaid Principal Balance (“UPB”)	$ 3,917,542	$ 3,671,468
Weighted Average Coupon	5.48%	5.13%
Weighted Average Term to Maturity (Months)	348 months	351 months
Weighted Average LTV Ratio	64.00%	65.00%
Weighted Average Original FICO	735	733
60+ Delinquency %	2.40%	2.60%
Settled Whole Loans | Non-QM loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 3,749,468	$ 3,520,671
Settled Whole Loans | Non-QM loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	72,290	56,825
Settled Whole Loans | Non-QM loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	24,934	32,253
Settled Whole Loans | Non-QM loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	70,850	61,719
Settled Whole Loans | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	1,745,417	1,411,997
Unpaid Principal Balance (“UPB”)	$ 1,759,641	$ 1,431,692
Weighted Average Coupon	8.45%	7.78%
Weighted Average Term to Maturity (Months)	11 months	12 months
Weighted Average LTV Ratio	65.00%	66.00%
Weighted Average Original FICO	744	746
60+ Delinquency %	4.10%	5.30%
Settled Whole Loans | Transitional loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 1,675,104	$ 1,348,815
Settled Whole Loans | Transitional loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	12,749	6,463
Settled Whole Loans | Transitional loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	8,188	2,234
Settled Whole Loans | Transitional loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	63,600	74,180
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	1,490,673	1,375,297
Unpaid Principal Balance (“UPB”)	$ 1,604,083	$ 1,485,967
Weighted Average Coupon	6.01%	5.74%
Weighted Average Term to Maturity (Months)	321 months	324 months
Weighted Average LTV Ratio	68.00%	69.00%
Weighted Average Original FICO	737	737
60+ Delinquency %	2.40%	2.40%
Settled Whole Loans | Single-family rental loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 1,555,427	$ 1,442,095
Settled Whole Loans | Single-family rental loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	10,893	8,431
Settled Whole Loans | Single-family rental loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	3,766	7,978
Settled Whole Loans | Single-family rental loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	33,997	27,463
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	75,347	82,884
Unpaid Principal Balance (“UPB”)	$ 82,695	$ 90,843
Weighted Average Coupon	4.01%	3.31%
Weighted Average Term to Maturity (Months)	147 months	151 months
Weighted Average LTV Ratio	29.00%	30.00%
Weighted Average Original FICO	726	714
60+ Delinquency %	3.80%	5.90%
Settled Whole Loans | Seasoned performing loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 78,138	$ 84,514
Settled Whole Loans | Seasoned performing loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,378	993
Settled Whole Loans | Seasoned performing loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	43	937
Settled Whole Loans | Seasoned performing loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	3,136	4,399
Settled Whole Loans | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	58,068	51,094
Unpaid Principal Balance (“UPB”)	$ 70,075	$ 61,816
Weighted Average Coupon	3.44%	3.44%
Weighted Average Term to Maturity (Months)	338 months	344 months
Weighted Average LTV Ratio	67.00%	68.00%
Weighted Average Original FICO	757	757
60+ Delinquency %	1.40%	0.00%
Settled Whole Loans | Agency eligible investor loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 69,082	$ 61,816
Settled Whole Loans | Agency eligible investor loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	0	0
Settled Whole Loans | Agency eligible investor loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	765	0
Settled Whole Loans | Agency eligible investor loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	228	0
Settled Whole Loans | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	428,157	448,887
Unpaid Principal Balance (“UPB”)	$ 528,843	$ 554,907
Weighted Average Coupon	4.75%	4.66%
Weighted Average Term to Maturity (Months)	272 months	277 months
Weighted Average LTV Ratio	62.00%	63.00%
60+ Delinquency %	16.70%	18.70%
Settled Whole Loans | Purchased Credit Deteriorated Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 394,413	$ 403,042
Settled Whole Loans | Purchased Credit Deteriorated Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	45,891	48,107
Settled Whole Loans | Purchased Credit Deteriorated Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	13,721	16,270
Settled Whole Loans | Purchased Credit Deteriorated Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	74,818	87,488
Settled Whole Loans | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	739,712	796,109
Unpaid Principal Balance (“UPB”)	$ 825,289	$ 884,257
Weighted Average Coupon	5.12%	5.01%
Weighted Average Term to Maturity (Months)	274 months	277 months
Weighted Average LTV Ratio	67.00%	68.00%
60+ Delinquency %	34.30%	39.60%
Settled Whole Loans | Purchased Non-Performing Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 448,977	$ 444,045
Settled Whole Loans | Purchased Non-Performing Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	93,267	89,623
Settled Whole Loans | Purchased Non-Performing Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	32,569	40,554
Settled Whole Loans | Purchased Non-Performing Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	250,476	310,035
Settled Whole Loans | Residential whole loans, total or weighted average
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	8,139,786	7,518,739
Unpaid Principal Balance (“UPB”)	$ 8,788,168	$ 8,180,950
Weighted Average Coupon	5.81%	5.64%
Weighted Average Term to Maturity (Months)	256 months	272 months
60+ Delinquency %	6.60%	8.10%
Settled Whole Loans | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	69.00%	70.00%
Transitional loans | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Securitization of eligible investor loans	$ 296,100	$ 223,200