Residential Whole Loans (Allowance for Credit Losses) (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2019	Mar. 31, 2019	Jun. 30, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	$ 218,011	$ 3,025	$ 1,651	$ 968	$ 3,025
Transition adjustment on adoption of ASU 2016-13		70,555			70,600
Current provision	(10,614)	74,897	385	683
Write-offs	(627)	(647)	(50)	0	(1,274)
Valuation adjustment on loans held for sale	(70,181)	70,181
Allowance for credit and valuation losses at March 31, 2020	136,589	218,011	1,986	1,651	136,589
Residential whole loans, at carrying value	4,813,119				4,813,119
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans					62,400
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	103,831	388	0	0	388
Transition adjustment on adoption of ASU 2016-13		6,904
Current provision	(2,297)	26,358	0	0
Write-offs	0	0	0	0
Valuation adjustment on loans held for sale	(70,181)	70,181
Allowance for credit and valuation losses at March 31, 2020	31,353	103,831	0	0	31,353
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	35,633	2,331	500	0	2,331
Transition adjustment on adoption of ASU 2016-13		517
Current provision	(5,213)	33,213	0	500
Write-offs	(420)	(428)	(50)	0
Valuation adjustment on loans held for sale	0	0
Allowance for credit and valuation losses at March 31, 2020	30,000	35,633	450	500	30,000
Residential whole loans, at carrying value	181,800				181,800
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	7,431	62	0	0	62
Transition adjustment on adoption of ASU 2016-13		754
Current provision	(500)	6,615	0	0
Write-offs	0	0	0	0
Valuation adjustment on loans held for sale	0	0
Allowance for credit and valuation losses at March 31, 2020	6,931	7,431	0	0	6,931
Seasoned performing loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	249	0	0	0	0
Transition adjustment on adoption of ASU 2016-13		19
Current provision	(25)	230	0	0
Write-offs	0	0	0	0
Valuation adjustment on loans held for sale	0	0
Allowance for credit and valuation losses at March 31, 2020	224	249	0	0	224
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit losses at December 31, 2019	70,867	244	1,151	968	244
Transition adjustment on adoption of ASU 2016-13		62,361
Current provision	(2,579)	8,481	385	183
Write-offs	(207)	(219)	0	0
Valuation adjustment on loans held for sale	0	0
Allowance for credit and valuation losses at March 31, 2020	68,081	$ 70,867	$ 1,536	$ 1,151	68,081
Residential whole loans, at carrying value	100,000				100,000
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit and valuation losses at March 31, 2020	2,100				2,100
Commitment to lend, unfunded	94,500				94,500
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Residential whole loans, at carrying value	$ 2,574,184				$ 2,574,184