Residential Whole Loans - Interest Income Components (Details) - Residential whole loans, total or weighted average - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 119,510	$ 99,466
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,138	9,009
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	97,576	69,731
Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	44,089	32,952
Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	28,227	14,861
Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	21,313	13,325
Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,090	1,010
Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	2,857	7,583
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	14,796	20,726
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	24,499	31,420
Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,138	9,009
Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	17,361	22,411
Carrying Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	12,674	13,141
Carrying Value | Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	620	3,567
Carrying Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	2,977	4,693
Carrying Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,090	1,010
Carrying Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0
Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	95,011	68,046
Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0
Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	80,215	47,320
Fair Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	31,415	19,811
Fair Value | Performing | Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	27,607	11,294
Fair Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	18,336	8,632
Fair Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0
Fair Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	2,857	7,583
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 14,796	$ 20,726