Residential Whole Loans (Allowance for Credit Losses) (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2021	Jan. 01, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 54,261	$ 63,244	$ 86,833	$ 136,589	$ 218,011	$ 3,025	$ 86,833
Transition adjustment on adoption of ASU 2016-13						70,555	70,600
Current provision	(9,626)	(8,583)	(22,372)	(30,062)	(10,614)	74,897
Write-offs	(533)	(400)	(1,217)	(281)	(627)	(647)	(2,149)
Valuation adjustment on loans held for sale					(70,181)	70,181
Ending balance	44,102	54,261	63,244	106,246	136,589	218,011	44,102
Loans	3,031,966						3,031,966
Allowance for loan loss	44,102	54,261	63,244	106,246	136,589	218,011	44,102
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans							62,400
Non-QM loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	12,092	14,545	21,068	31,353	103,831	388	21,068
Transition adjustment on adoption of ASU 2016-13						6,904
Current provision	(2,403)	(2,416)	(6,523)	(4,568)	(2,297)	26,358
Write-offs	0	(37)	0	(32)	0	0
Valuation adjustment on loans held for sale					(70,181)	70,181
Ending balance	9,689	12,092	14,545	26,753	31,353	103,831	9,689
Allowance for loan loss	9,689	12,092	14,545	26,753	31,353	103,831	9,689
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	11,604	13,668	18,371	30,000	35,633	2,331	18,371
Transition adjustment on adoption of ASU 2016-13						517
Current provision	(2,526)	(1,809)	(3,700)	(7,140)	(5,213)	33,213
Write-offs	(393)	(255)	(1,003)	(227)	(420)	(428)
Valuation adjustment on loans held for sale					0	0
Ending balance	8,685	11,604	13,668	22,633	30,000	35,633	8,685
Loans	94,900			143,400			94,900
Allowance for loan loss	8,685	11,604	13,668	22,633	30,000	35,633	8,685
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	2,360	2,746	3,918	6,931	7,431	62	3,918
Transition adjustment on adoption of ASU 2016-13						754
Current provision	(670)	(386)	(1,172)	(1,906)	(500)	6,615
Write-offs	(56)	0	0	0	0	0
Valuation adjustment on loans held for sale					0	0
Ending balance	1,634	2,360	2,746	5,025	6,931	7,431	1,634
Allowance for loan loss	1,634	2,360	2,746	5,025	6,931	7,431	1,634
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	57	66	107	224	249	0	107
Transition adjustment on adoption of ASU 2016-13						19
Current provision	(7)	(9)	(41)	(74)	(25)	230
Write-offs	0	0	0	0	0	0
Valuation adjustment on loans held for sale					0	0
Ending balance	50	57	66	150	224	249	50
Allowance for loan loss	50	57	66	150	224	249	50
Purchased credit deteriorated loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	28,148	32,219	43,369	68,081	70,867	244	43,369
Transition adjustment on adoption of ASU 2016-13						62,361
Current provision	(4,020)	(3,963)	(10,936)	(16,374)	(2,579)	8,481
Write-offs	(84)	(108)	(214)	(22)	(207)	(219)
Valuation adjustment on loans held for sale					0	0
Ending balance	24,044	28,148	32,219	51,685	68,081	70,867	24,044
Loans	57,400			72,700			57,400
Allowance for loan loss	24,044	$ 28,148	$ 32,219	51,685	$ 68,081	$ 70,867	24,044
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Ending balance	355			1,600			355
Commitment to lend, unfunded	29,200			73,200			29,200
Allowance for loan loss	$ 355			$ 1,600			$ 355
Cumulative Effect, Period of Adoption, Adjustment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for loan loss								$ 8,300