Residential Whole Loans - Narrative (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2023 USD ($) loan	Dec. 31, 2022 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total residential whole loans	$ 9,000,000	$ 7,500,000
Interest income on nonaccrual loans	$ 14,900
Number of loan modifications | loan	4
Average increase in weighted life	23 months
Loan carrying value	$ 563
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of loan modifications | loan	1
Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of loan modifications | loan	3
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 315,400	413,100
Interest income on nonaccrual loans	9,800
Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale		337,800
Loss on sale of financing receivable		72,300
Amount derecognized		598,000
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale		101,200
Loss on sale of financing receivable		26,400
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	266,900	195,100
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	66,500	80,500
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 51,600	$ 71,700