Residential Whole Loans - Residential Whole Loans, at Carrying Value, Additional Information (Details) - USD ($) $ in Thousands	3 Months Ended	6 Months Ended	12 Months Ended
	Mar. 31, 2024	Jun. 30, 2023	Dec. 31, 2023
Accounts, Notes, Loans and Financing Receivable [Line Items]
LTV ratio (as a percent)	80.00%
Loans	$ 1,484,370
Residential whole loans, total or weighted average
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans, fair value	14,800		$ 103,700
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	816,617
Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	29,098
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 66,065
Multi-Family Transitional Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
LTV ratio (as a percent)	63.00%	63.00%
Loans	$ 1,200,000		1,200,000
Weighted Average Term to Maturity (Months)	12 months	14 months
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	$ 8,024,799		7,814,106
Unpaid Principal Balance (“UPB”)	$ 8,366,801		$ 8,175,387
Weighted Average Coupon ( as a percent)	7.11%		6.86%
Weighted Average Term to Maturity (Months)	238 months		240 months
60+ Delinquency %	4.30%		3.80%
Settled Whole Loans | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	$ 3,836,705		$ 3,700,052
Unpaid Principal Balance (“UPB”)	$ 4,059,991		$ 3,934,798
Weighted Average Coupon ( as a percent)	6.02%		5.78%
Weighted Average Term to Maturity (Months)	342 months		344 months
LTV ratio (as a percent)	65.00%		65.00%
Weighted Average Original FICO	734		735
60+ Delinquency %	3.20%		2.70%
Settled Whole Loans | Non-QM loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 3,814,533		$ 3,732,327
Settled Whole Loans | Non-QM loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	115,484		98,017
Settled Whole Loans | Non-QM loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	41,428		29,587
Settled Whole Loans | Non-QM loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	88,546		74,867
Settled Whole Loans | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	2,493,073		2,358,909
Unpaid Principal Balance (“UPB”)	$ 2,502,067		$ 2,368,121
Weighted Average Coupon ( as a percent)	9.45%		9.22%
Weighted Average Term to Maturity (Months)	9 months		10 months
LTV ratio (as a percent)	64.00%		64.00%
Weighted Average Original FICO	747		747
60+ Delinquency %	6.00%		5.10%
Settled Whole Loans | Transitional loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 2,306,508		$ 2,187,161
Settled Whole Loans | Transitional loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	44,621		61,024
Settled Whole Loans | Transitional loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	18,459		26,618
Settled Whole Loans | Transitional loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	132,479		93,318
Settled Whole Loans | Single-family Rental Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	1,574,322		1,630,442
Unpaid Principal Balance (“UPB”)	$ 1,665,788		$ 1,729,923
Weighted Average Coupon ( as a percent)	6.52%		6.30%
Weighted Average Term to Maturity (Months)	331 months		320 months
LTV ratio (as a percent)	69.00%		70.00%
Weighted Average Original FICO	738		738
60+ Delinquency %	4.60%		4.70%
Settled Whole Loans | Single-family Rental Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 1,571,772		$ 1,636,810
Settled Whole Loans | Single-family Rental Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	17,395		12,543
Settled Whole Loans | Single-family Rental Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	6,452		12,314
Settled Whole Loans | Single-family Rental Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	70,169		68,256
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	66,045		68,924
Unpaid Principal Balance (“UPB”)	$ 72,658		$ 75,715
Weighted Average Coupon ( as a percent)	4.77%		4.58%
Weighted Average Term to Maturity (Months)	140 months		143 months
LTV ratio (as a percent)	28.00%		28.00%
Weighted Average Original FICO	725		725
60+ Delinquency %	1.90%		3.40%
Settled Whole Loans | Seasoned performing loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 70,016		$ 72,126
Settled Whole Loans | Seasoned performing loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,271		1,045
Settled Whole Loans | Seasoned performing loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	43		235
Settled Whole Loans | Seasoned performing loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,328		2,309
Settled Whole Loans | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	54,654		55,779
Unpaid Principal Balance (“UPB”)	$ 66,297		$ 66,830
Weighted Average Coupon ( as a percent)	3.44%		3.44%
Weighted Average Term to Maturity (Months)	329 months		332 months
LTV ratio (as a percent)	66.00%		66.00%
Weighted Average Original FICO	757		758
60+ Delinquency %	1.10%		0.30%
Settled Whole Loans | Agency eligible investor loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 65,064		$ 65,094
Settled Whole Loans | Agency eligible investor loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	523		1,508
Settled Whole Loans | Agency eligible investor loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	223		0
Settled Whole Loans | Agency eligible investor loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	487		228
Settled Whole Loans | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	412,077		418,109
Unpaid Principal Balance (“UPB”)	$ 499,761		$ 506,828
Weighted Average Coupon ( as a percent)	4.85%		4.83%
Weighted Average Term to Maturity (Months)	265 months		267 months
LTV ratio (as a percent)	58.00%		59.00%
60+ Delinquency %	15.90%		16.20%
Settled Whole Loans | Purchased Credit Deteriorated Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 373,341		$ 379,970
Settled Whole Loans | Purchased Credit Deteriorated Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	46,972		44,731
Settled Whole Loans | Purchased Credit Deteriorated Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	16,784		12,814
Settled Whole Loans | Purchased Credit Deteriorated Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	62,664		69,313
Settled Whole Loans | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	681,789		705,424
Unpaid Principal Balance (“UPB”)	$ 753,035		$ 772,737
Weighted Average Coupon ( as a percent)	5.24%		5.21%
Weighted Average Term to Maturity (Months)	268 months		270 months
LTV ratio (as a percent)	60.00%		62.00%
60+ Delinquency %	29.90%		30.00%
Settled Whole Loans | Purchased Non-Performing Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 437,507		$ 444,491
Settled Whole Loans | Purchased Non-Performing Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	90,223		96,464
Settled Whole Loans | Purchased Non-Performing Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	31,434		31,560
Settled Whole Loans | Purchased Non-Performing Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	193,871		200,222
Settled Whole Loans | Residential whole loans, total or weighted average
Accounts, Notes, Loans and Financing Receivable [Line Items]
Fair Value / Carrying Value	9,118,665		8,937,639
Unpaid Principal Balance (“UPB”)	$ 9,619,597		$ 9,454,952
Weighted Average Coupon ( as a percent)	6.21%		6.04%
Weighted Average Term to Maturity (Months)	227 months		234 months
60+ Delinquency %	6.90%		6.60%
Settled Whole Loans | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
LTV ratio (as a percent)	67.00%		68.00%
Transitional loans | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Securitization of eligible investor loans	$ 608,900		$ 551,300