Residential Whole Loans (Allowance for Credit Losses) (Details) - USD ($)	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020	Dec. 31, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 86,833,000	$ 3,025,000
Transition adjustment on adoption of ASU 2016-13		70,555,000
Current provision	(22,372,000)	74,897,000
Write-offs	(1,217,000)	(647,000)
Valuation adjustment on loans held for sale		(70,181,000)
Ending balance	63,244,000	218,011,000
Residential whole loans, at carrying value	3,932,300,000
Fair  Value	350,115,000		$ 399,999,000
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans		62,400,000
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	21,068,000	388,000
Transition adjustment on adoption of ASU 2016-13		6,904,000
Current provision	(6,523,000)	26,358,000
Write-offs	0	0
Valuation adjustment on loans held for sale		(70,181,000)
Ending balance	14,545,000	103,831,000
Fair  Value	895,300,000
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	18,371,000	2,331,000
Transition adjustment on adoption of ASU 2016-13		517,000
Current provision	(3,700,000)	33,213,000
Write-offs	(1,003,000)	(428,000)
Valuation adjustment on loans held for sale		0
Ending balance	13,668,000	35,633,000
Residential whole loans, at carrying value	149,200,000	110,800,000
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	3,918,000	62,000
Transition adjustment on adoption of ASU 2016-13		754,000
Current provision	(1,172,000)	6,615,000
Write-offs	0	0
Valuation adjustment on loans held for sale		0
Ending balance	2,746,000	7,431,000
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	107,000	0
Transition adjustment on adoption of ASU 2016-13		19,000
Current provision	(41,000)	230,000
Write-offs	0	0
Valuation adjustment on loans held for sale		0
Ending balance	66,000	249,000
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	43,369,000	244,000
Transition adjustment on adoption of ASU 2016-13		62,361,000
Current provision	(10,936,000)	8,481,000
Write-offs	(214,000)	(219,000)
Valuation adjustment on loans held for sale		0
Ending balance	32,219,000	70,867,000
Residential whole loans, at carrying value	87,700,000	74,500,000
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	795,905	3,500,000
Commitment to lend, unfunded	54,400,000	$ 123,100,000
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs	0
Residential whole loans, at carrying value	$ 2,243,444,000