Residential Whole Loans (Allowance for Credit Losses) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 106,246	$ 136,589	$ 218,011	$ 3,025	$ 2,271	$ 1,986	$ 1,651	$ 968	$ 3,025
Transition adjustment on adoption of ASU 2016-13				70,555					70,600
Current provision/(reversal)	(18,583)	(30,062)	(10,614)	74,897	1,031	347	385	683
Write-offs	(830)	(281)	(627)	(647)	(277)	(62)	(50)	0	(2,385)
Valuation adjustment on loans held for sale			(70,181)	70,181
Ending balance	86,833	106,246	136,589	218,011	3,025	2,271	1,986	1,651	86,833
Residential whole loans, at carrying value	4,195,332								4,195,332
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans									62,400
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	26,753	31,353	103,831	388	0	0	0	0	388
Transition adjustment on adoption of ASU 2016-13				6,904
Current provision/(reversal)	(5,599)	(4,568)	(2,297)	26,358	388	0	0	0
Write-offs	(86)	(32)	0	0	0	0	0	0
Valuation adjustment on loans held for sale			(70,181)	70,181
Ending balance	21,068	26,753	31,353	103,831	388	0	0	0	21,068
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	22,633	30,000	35,633	2,331	388	450	500	0	2,331
Transition adjustment on adoption of ASU 2016-13				517
Current provision/(reversal)	(3,837)	(7,140)	(5,213)	33,213	2,220	0	0	500
Write-offs	(425)	(227)	(420)	(428)	(277)	(62)	(50)	0
Valuation adjustment on loans held for sale			0	0
Ending balance	18,371	22,633	30,000	35,633	2,331	388	450	500	18,371
Residential whole loans, at carrying value	161,800								161,800
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	5,025	6,931	7,431	62	0	0	0	0	62
Transition adjustment on adoption of ASU 2016-13				754
Current provision/(reversal)	(1,107)	(1,906)	(500)	6,615	62	0	0	0
Write-offs	0	0	0	0	0	0	0	0
Valuation adjustment on loans held for sale			0	0
Ending balance	3,918	5,025	6,931	7,431	62	0	0	0	3,918
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	150	224	249	0	0	0	0	0	0
Transition adjustment on adoption of ASU 2016-13				19
Current provision/(reversal)	(43)	(74)	(25)	230	0	0	0	0
Write-offs	0	0	0	0	0	0	0	0
Valuation adjustment on loans held for sale			0	0
Ending balance	107	150	224	249	0	0	0	0	107
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	51,685	68,081	70,867	244	1,883	1,536	1,151	968	244
Transition adjustment on adoption of ASU 2016-13				62,361
Current provision/(reversal)	(7,997)	(16,374)	(2,579)	8,481	(1,639)	347	385	183
Write-offs	(319)	(22)	(207)	(219)	0	0	0	0
Valuation adjustment on loans held for sale			0	0
Ending balance	43,369	$ 51,685	$ 68,081	$ 70,867	$ 244	$ 1,883	$ 1,536	$ 1,151	43,369
Residential whole loans, at carrying value	70,300								70,300
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	1,200								1,200
Commitment to lend, unfunded	60,600								60,600
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									(117)
Residential whole loans, at carrying value	2,357,185								2,357,185
Cumulative Effect, Period of Adoption, Adjustment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	$ 8,300								$ 8,300