Residential Whole Loans - Narrative (Details) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2024 USD ($) loan	Dec. 31, 2023 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total residential whole loans	$ 9,100,000	$ 9,000,000
Interest income	$ 3,300
Number of loan modifications | loan	1	5
Average increase in weighted life	29 months	24 months
Change in carrying value of the loan	$ 100
Loan carrying value	600
Non-QM and Single-Family Rental Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	171,000
Loss on sale of financing receivable	21,700
Reversal of previously recorded unrealized losses	23,700
Gain from sale of loans	$ 2,000
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of loan modifications | loan	1
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 298,600	$ 315,400
Interest income	2,200
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	317,300	266,900
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	61,200	66,500
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 51,000	$ 51,600