Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($)	3 Months Ended
	Mar. 31, 2022	Mar. 31, 2021
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 39,447,000	$ 86,833,000
Current provision	(3,467,000)	(22,372,000)
Write-offs	(523,000)	(1,217,000)
Ending balance	35,457,000	63,244,000
Allowance for loan loss	35,457,000	63,244,000
Loans	2,320,048,000
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	8,289,000	21,068,000
Current provision	(909,000)	(6,523,000)
Write-offs	(51,000)	0
Ending balance	7,329,000	14,545,000
Allowance for loan loss	7,329,000	14,545,000
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	6,881,000	18,371,000
Current provision	(1,460,000)	(3,700,000)
Write-offs	(219,000)	(1,003,000)
Ending balance	5,202,000	13,668,000
Allowance for loan loss	5,202,000	13,668,000
Loans	80,200,000	149,200,000
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,451,000	3,918,000
Current provision	(122,000)	(1,172,000)
Write-offs	(27,000)	0
Ending balance	1,302,000	2,746,000
Allowance for loan loss	1,302,000	2,746,000
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	46,000	107,000
Current provision	(1,000)	(41,000)
Write-offs	0	0
Ending balance	45,000	66,000
Allowance for loan loss	45,000	66,000
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	22,780,000	43,369,000
Current provision	(975,000)	(10,936,000)
Write-offs	(226,000)	(214,000)
Ending balance	21,579,000	32,219,000
Allowance for loan loss	21,579,000	32,219,000
Loans	69,100,000	87,700,000
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	156,000	795,905
Commitment to lend, unfunded	12,900,000	54,400,000
Allowance for loan loss	$ 156,000	$ 795,905