Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2023	Mar. 31, 2023	Jun. 30, 2022	Mar. 31, 2022	Jun. 30, 2023
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 33,062	$ 35,314	$ 35,457	$ 39,447	$ 35,314
Current provision/(reversal)	265	315	1,830	(3,467)
Write-offs	(2,292)	(2,567)	(360)	(523)	(4,859)
Ending balance	31,035	33,062	36,927	35,457	31,035
Allowance for loan loss	31,035	33,062	36,927	35,457	31,035
Loans	1,670,788				1,670,788
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	7,145	7,359	7,329	8,289	7,359
Current provision/(reversal)	(233)	(214)	(199)	(909)
Write-offs	(206)	0	0	(51)
Ending balance	6,706	7,145	7,130	7,329	6,706
Allowance for loan loss	6,706	7,145	7,130	7,329	6,706
Transitional loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	3,626	5,223	5,202	6,881	5,223
Current provision/(reversal)		406		(1,460)
Write-offs		(2,003)		(219)
Ending balance		3,626		5,202
Allowance for loan loss		3,626		5,202
Loans	35,200		71,900		35,200
Transitional loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Current provision/(reversal)	999		(23)
Write-offs	(1,785)		(118)
Ending balance	2,840		5,061		2,840
Allowance for loan loss	2,840		5,061		2,840
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,340	1,277	1,302	1,451	1,277
Current provision/(reversal)	(103)	514	174	(122)
Write-offs	0	(451)	(184)	(27)
Ending balance	1,237	1,340	1,292	1,302	1,237
Allowance for loan loss	1,237	1,340	1,292	1,302	1,237
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	46	48	45	46	48
Current provision/(reversal)	(4)	(2)	1	(1)
Write-offs	0	0	0	0
Ending balance	42	46	46	45	42
Allowance for loan loss	42	46	46	45	42
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	20,905	21,407	21,579	22,780	21,407
Current provision/(reversal)	(394)	(389)	1,877	(975)
Write-offs	(301)	(113)	(58)	(226)
Ending balance	20,210	20,905	23,398	21,579	20,210
Allowance for loan loss	20,210	$ 20,905	23,398	$ 21,579	20,210
Loans	55,500		63,800		55,500
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	44		110		44
Commitment to lend, unfunded	6,000		10,100		6,000
Allowance for loan loss	$ 44		$ 110		$ 44