Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2022	Sep. 30, 2022	Jun. 30, 2022	Mar. 31, 2022	Dec. 31, 2021	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2022
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 37,192	$ 36,927	$ 35,457	$ 39,447	$ 44,102	$ 54,261	$ 63,244	$ 86,833	$ 39,447
Current provision	(1,576)	547	1,830	(3,467)	(3,430)	(9,626)	(8,583)	(22,372)
Write-offs	(302)	(282)	(360)	(523)	(1,225)	(533)	(400)	(1,217)	(1,470)
Ending balance	35,314	37,192	36,927	35,457	39,447	44,102	54,261	63,244	35,314
Loans	1,826,529								1,826,529
Allowance for credit loss	35,314	37,192	36,927	35,457	39,447	44,102	54,261	63,244	35,314
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	6,888	7,130	7,329	8,289	9,689	12,092	14,545	21,068	8,289
Current provision	471	(242)	(199)	(909)	(1,400)	(2,403)	(2,416)	(6,523)
Write-offs	0	0	0	(51)	0	0	(37)	0
Ending balance	7,359	6,888	7,130	7,329	8,289	9,689	12,092	14,545	7,359
Allowance for credit loss	7,359	6,888	7,130	7,329	8,289	9,689	12,092	14,545	7,359
Transitional loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	5,530	5,061	5,202	6,881	8,685	11,604	13,668	18,371	6,881
Current provision	(13)	583	(23)	(1,460)	(706)	(2,526)	(1,809)	(3,700)
Write-offs	(294)	(114)	(118)	(219)	(1,098)	(393)	(255)	(1,003)
Ending balance	5,223	5,530	5,061	5,202	6,881	8,685	11,604	13,668	5,223
Loans	56,100				87,000				56,100
Allowance for credit loss	5,223	5,530	5,061	5,202	6,881	8,685	11,604	13,668	5,223
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,314	1,292	1,302	1,451	1,634	2,360	2,746	3,918	1,451
Current provision	(37)	83	174	(122)	(178)	(670)	(386)	(1,172)
Write-offs	0	(61)	(184)	(27)	(5)	(56)	0	0
Ending balance	1,277	1,314	1,292	1,302	1,451	1,634	2,360	2,746	1,277
Allowance for credit loss	1,277	1,314	1,292	1,302	1,451	1,634	2,360	2,746	1,277
Seasoned Performing Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	49	46	45	46	50	57	66	107	46
Current provision	(1)	3	1	(1)	(4)	(7)	(9)	(41)
Write-offs	0	0	0	0	0	0	0	0
Ending balance	48	49	46	45	46	50	57	66	48
Allowance for credit loss	48	49	46	45	46	50	57	66	48
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	23,411	23,398	21,579	22,780	24,044	28,148	32,219	43,369	22,780
Current provision	(1,996)	120	1,877	(975)	(1,142)	(4,020)	(3,963)	(10,936)
Write-offs	(8)	(107)	(58)	(226)	(122)	(84)	(108)	(214)
Ending balance	21,407	23,411	23,398	21,579	22,780	24,044	28,148	32,219	21,407
Loans	48,500				57,400				48,500
Allowance for credit loss	21,407	$ 23,411	$ 23,398	21,579	22,780	$ 24,044	$ 28,148	$ 32,219	21,407
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance				$ 205					205
Ending balance	29				205				29
Commitment to lend, unfunded	8,000				18,500				8,000
Allowance for credit loss	$ 29				$ 205				$ 29