Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2024	Mar. 31, 2023
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 20,451	$ 35,314
Current provision/(reversal)	(460)	315
Write-offs	(379)
Write-offs	(379)	(2,567)
Ending balance	19,612	33,062
Allowance for loan loss	19,612	33,062
Loans	1,484,370
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,870	7,359
Current provision/(reversal)	(189)	(214)
Write-offs	0
Write-offs		0
Ending balance	1,681	7,145
Allowance for loan loss	1,681	7,145
Transitional loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	2,588	5,223
Current provision/(reversal)	(473)	406
Write-offs	(416)
Write-offs		(2,003)
Ending balance	1,699	3,626
Allowance for loan loss	1,699	3,626
Loans	21,900	46,400
Single-family Rental Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	4,355	1,277
Current provision/(reversal)	228	514
Write-offs	59
Write-offs		(451)
Ending balance	4,642	1,340
Allowance for loan loss	4,642	1,340
Loans	10,600
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	21	48
Current provision/(reversal)	(1)	(2)
Write-offs	0
Write-offs		0
Ending balance	20	46
Allowance for loan loss	20	46
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	11,617	21,407
Current provision/(reversal)	(25)	(389)
Write-offs	(22)
Write-offs		(113)
Ending balance	11,570	20,905
Allowance for loan loss	11,570	20,905
Loans	50,500	62,000
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	0	16
Commitment to lend, unfunded	2,000	6,800
Allowance for loan loss	$ 0	$ 16