Residential Whole Loans - Interest Income Components (Details) - Residential Whole Loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 537,883	$ 441,223	$ 303,468
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	445,773	329,276	182,597
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	62,464	78,520	80,741
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	193,327	149,743	96,948
Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	149,429	75,524	33,129
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	94,141	68,975	34,169
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,504	4,673	6,684
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,372	30,361	11,667
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	29,646	33,427	40,130
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	94,134	112,583	169,618
Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	64,488	79,156	129,488
Held at Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Carrying Value | Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	47,471	51,359	75,517
Held at Carrying Value | Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,346	7,810	22,424
Held at Carrying Value | Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	11,167	15,314	24,863
Held at Carrying Value | Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,504	4,673	6,684
Held at Carrying Value | Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	29,646	33,427	40,130
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	443,749	328,640	133,850
Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	381,285	250,120	53,109
Held at Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	62,464	78,520	80,741
Held at Fair Value | Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	145,856	98,384	21,431
Held at Fair Value | Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	148,083	67,714	10,705
Held at Fair Value | Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	82,974	53,661	9,306
Held at Fair Value | Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Fair Value | Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,372	30,361	11,667
Held at Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 0	$ 0	$ 0