Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2022	Jun. 30, 2022	Mar. 31, 2022	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Sep. 30, 2022
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 36,927	$ 35,457	$ 39,447	$ 54,261	$ 63,244	$ 86,833	$ 39,447
Current provision/(reversal)	547	1,830	(3,467)	(9,626)	(8,583)	(22,372)
Write-offs	(282)	(360)	(523)	(533)	(400)	(1,217)	(1,165)
Ending balance	37,192	36,927	35,457	44,102	54,261	63,244	37,192
Allowance for loan loss	37,192	36,927	35,457	44,102	54,261	63,244	37,192
Loans	1,915,277						1,915,277
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	7,130	7,329	8,289	12,092	14,545	21,068	8,289
Current provision/(reversal)	(242)	(199)	(909)	(2,403)	(2,416)	(6,523)
Write-offs	0	0	(51)		(37)	0
Ending balance	6,888	7,130	7,329	9,689	12,092	14,545	6,888
Allowance for loan loss	6,888	7,130	7,329	9,689	12,092	14,545	6,888
Residential transition loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	5,061	5,202	6,881	11,604	13,668	18,371	6,881
Current provision/(reversal)	583	(23)	(1,460)	(2,526)	(1,809)	(3,700)
Write-offs	(114)	(118)	(219)	(393)	(255)	(1,003)
Ending balance	5,530	5,061	5,202	8,685	11,604	13,668	5,530
Allowance for loan loss	5,530	5,061	5,202	8,685	11,604	13,668	5,530
Loans	66,700			94,900			66,700
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,292	1,302	1,451	2,360	2,746	3,918	1,451
Current provision/(reversal)	83	174	(122)	(670)	(386)	(1,172)
Write-offs	(61)	(184)	(27)	(56)	0	0
Ending balance	1,314	1,292	1,302	1,634	2,360	2,746	1,314
Allowance for loan loss	1,314	1,292	1,302	1,634	2,360	2,746	1,314
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	46	45	46	57	66	107	46
Current provision/(reversal)	3	1	(1)	(7)	(9)	(41)
Write-offs	0	0	0	0	0	0
Ending balance	49	46	45	50	57	66	49
Allowance for loan loss	49	46	45	50	57	66	49
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	23,398	21,579	22,780	28,148	32,219	43,369	22,780
Current provision/(reversal)	120	1,877	(975)	(4,020)	(3,963)	(10,936)
Write-offs	(107)	(58)	(226)	(84)	(108)	(214)
Ending balance	23,411	23,398	21,579	24,044	28,148	32,219	23,411
Allowance for loan loss	23,411	$ 23,398	$ 21,579	24,044	$ 28,148	$ 32,219	23,411
Loans	56,200			57,400			56,200
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	84			355			84
Commitment to lend, unfunded	8,400			29,200			8,400
Allowance for loan loss	$ 84			$ 355			$ 84