Residential Whole Loans (Allowance for Credit Losses) (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Sep. 30, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 136,589	$ 218,011	$ 3,025	$ 1,986	$ 1,651	$ 968	$ 3,025
Transition adjustment on adoption of ASU 2016-13			70,555				70,600
Current provision	(30,062)	(10,614)	74,897	347	385	683
Write-offs	(281)	(627)	(647)	(62)	(50)	0	(1,555)
Valuation adjustment on loans held for sale		(70,181)	70,181
Ending balance	106,246	136,589	218,011	2,271	1,986	1,651	106,246
Residential whole loans, at carrying value	4,493,806						4,493,806
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans							62,400
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	31,353	103,831	388	0	0	0	388
Transition adjustment on adoption of ASU 2016-13			6,904
Current provision	(4,568)	(2,297)	26,358	0	0	0
Write-offs	(32)	0	0	0	0	0
Valuation adjustment on loans held for sale		(70,181)	70,181
Ending balance	26,753	31,353	103,831	0	0	0	26,753
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	30,000	35,633	2,331	450	500	0	2,331
Transition adjustment on adoption of ASU 2016-13			517
Current provision	(7,140)	(5,213)	33,213	0	0	500
Write-offs	(227)	(420)	(428)	(62)	(50)	0
Valuation adjustment on loans held for sale		0	0
Ending balance	22,633	30,000	35,633	388	450	500	22,633
Residential whole loans, at carrying value	143,400						143,400
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	6,931	7,431	62	0	0	0	62
Transition adjustment on adoption of ASU 2016-13			754
Current provision	(1,906)	(500)	6,615	0	0	0
Write-offs	0	0	0	0	0	0
Valuation adjustment on loans held for sale		0	0
Ending balance	5,025	6,931	7,431	0	0	0	5,025
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	224	249	0	0	0	0	0
Transition adjustment on adoption of ASU 2016-13			19
Current provision	(74)	(25)	230	0	0	0
Write-offs	0	0	0	0	0	0
Valuation adjustment on loans held for sale		0	0
Ending balance	150	224	249	0	0	0	150
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	68,081	70,867	244	1,536	1,151	968	244
Transition adjustment on adoption of ASU 2016-13			62,361
Current provision	(16,374)	(2,579)	8,481	347	385	183
Write-offs	(22)	(207)	(219)	0	0	0
Valuation adjustment on loans held for sale		0	0
Ending balance	51,685	$ 68,081	$ 70,867	$ 1,883	$ 1,536	$ 1,151	51,685
Residential whole loans, at carrying value	72,700						72,700
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	1,600						1,600
Commitment to lend, unfunded	73,200						73,200
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs							(32)
Residential whole loans, at carrying value	$ 2,465,148						$ 2,465,148