Residential Whole Loans - Interest Income Components (Details) - Residential whole loans, total or weighted average - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2023	Sep. 30, 2022	Sep. 30, 2023	Sep. 30, 2022
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 140,538	$ 114,415	$ 388,096	$ 316,235
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,371	7,916	22,596	25,597
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	117,615	87,767	318,116	232,370
Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	51,724	40,658	141,330	108,121
Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	40,223	19,342	101,071	49,389
Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	24,087	18,998	68,543	48,737
Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,095	1,227	3,312	3,392
Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	486	7,542	3,860	22,731
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	15,552	18,732	47,384	58,268
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	23,582	27,231	72,340	85,785
Held at Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,371	7,916	22,596	25,597
Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	16,211	19,315	49,744	60,188
Held at Carrying Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	12,276	13,646	36,595	39,041
Held at Carrying Value | Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	101	784	1,239	5,621
Held at Carrying Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	2,739	3,658	8,598	12,134
Held at Carrying Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,095	1,227	3,312	3,392
Held at Carrying Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Held at Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	116,956	87,184	315,756	230,450
Held at Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	101,404	68,452	268,372	172,182
Held at Fair Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	39,448	27,012	104,735	69,080
Held at Fair Value | Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	40,122	18,558	99,832	43,768
Held at Fair Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	21,348	15,340	59,945	36,603
Held at Fair Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Held at Fair Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	486	7,542	3,860	22,731
Held at Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 15,552	$ 18,732	$ 47,384	$ 58,268