Residential Whole Loans - Narrative (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2022 USD ($) loan	Dec. 31, 2020 USD ($)	Dec. 31, 2021 USD ($)
Receivables [Abstract]
Total residential whole loans	$ 7,500,000		$ 7,900,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable number of whole loans sold | loan	0
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loss on sale of financing receivable	$ 72,300
Financing receivable, nonaccrual	413,100		588,100
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale		$ 1,800,000
Loss on sale of financing receivable		273,000
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	195,100		240,200
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	80,500		108,900
Interest Income On Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	18,800		13,200
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	71,700		$ 107,400
Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	337,800
Agency eligible investor loans | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	$ 598,000	24,100
Loss on sale of financing receivable		$ 800