Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 35,314	$ 39,447
Current provision	315	(3,467)
Write-offs	(2,567)	(523)
Ending balance	33,062	35,457
Allowance for loan loss	33,062	35,457
Loans	1,753,080
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	7,359	8,289
Current provision	(214)	(909)
Write-offs	0	(51)
Ending balance	7,145	7,329
Allowance for loan loss	7,145	7,329
Transitional loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	5,223	6,881
Current provision	406	(1,460)
Write-offs	(2,003)	(219)
Ending balance	3,626	5,202
Allowance for loan loss	3,626	5,202
Loans	46,400	80,200
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,277	1,451
Current provision	514	(122)
Write-offs	(451)	(27)
Ending balance	1,340	1,302
Allowance for loan loss	1,340	1,302
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	48	46
Current provision	(2)	(1)
Write-offs	0	0
Ending balance	46	45
Allowance for loan loss	46	45
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	21,407	22,780
Current provision	(389)	(975)
Write-offs	(113)	(226)
Ending balance	20,905	21,579
Allowance for loan loss	20,905	21,579
Loans	62,000	69,100
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	16	156
Commitment to lend, unfunded	6,800	12,900
Allowance for loan loss	$ 16	$ 156