Residential Whole Loans - Residential Whole Loans, at Carrying Value Additional Information (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2022	Dec. 31, 2021
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Unsettled residential whole loans	$ 28,900
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Purchased Performing Loans	6,748,331	$ 6,315,738
Unpaid Principal Balance (“UPB”)	$ 6,869,096	$ 6,189,012
Weighted Average Coupon	5.00%	5.05%
Weighted Average Term to Maturity (Months)	302 months	307 months
Settled Whole Loans | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 3,621,124	$ 3,453,242
Unpaid Principal Balance (“UPB”)	$ 3,670,937	$ 3,361,164
Weighted Average Coupon	4.92%	5.07%
Weighted Average Term to Maturity (Months)	356 months	355 months
Weighted Average LTV Ratio	65.00%	66.00%
Weighted Average FICO Scores	733	731
Settled Whole Loans | Non-QM loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 3,431,011	$ 3,165,964
Settled Whole Loans | Non-QM loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	119,445	77,581
Settled Whole Loans | Non-QM loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	30,355	22,864
Settled Whole Loans | Non-QM loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	90,126	94,755
Settled Whole Loans | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	885,155	727,964
Unpaid Principal Balance (“UPB”)	$ 886,942	$ 731,154
Weighted Average Coupon	7.11%	7.18%
Weighted Average Term to Maturity (Months)	12 months	11 months
Weighted Average LTV Ratio	67.00%	67.00%
Weighted Average FICO Scores	740	735
Settled Whole Loans | Rehabilitation loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 783,366	$ 616,733
Settled Whole Loans | Rehabilitation loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	14,118	5,834
Settled Whole Loans | Rehabilitation loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	3,178	5,553
Settled Whole Loans | Rehabilitation loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	86,280	103,034
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	1,152,195	949,772
Unpaid Principal Balance (“UPB”)	$ 1,168,778	$ 924,498
Weighted Average Coupon	5.26%	5.46%
Weighted Average Term to Maturity (Months)	324 months	329 months
Weighted Average LTV Ratio	70.00%	70.00%
Weighted Average FICO Scores	735	732
Settled Whole Loans | Single-family rental loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 1,133,996	$ 898,166
Settled Whole Loans | Single-family rental loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	13,436	2,150
Settled Whole Loans | Single-family rental loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	547	695
Settled Whole Loans | Single-family rental loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	20,799	23,487
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	98,224	101,995
Unpaid Principal Balance (“UPB”)	$ 107,624	$ 111,710
Weighted Average Coupon	2.71%	2.76%
Weighted Average Term to Maturity (Months)	159 months	162 months
Weighted Average LTV Ratio	36.00%	37.00%
Weighted Average FICO Scores	722	722
Settled Whole Loans | Seasoned performing loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 98,569	$ 102,047
Settled Whole Loans | Seasoned performing loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	634	938
Settled Whole Loans | Seasoned performing loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	97	481
Settled Whole Loans | Seasoned performing loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	8,324	8,244
Settled Whole Loans | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	991,633	1,082,765
Unpaid Principal Balance (“UPB”)	$ 1,034,815	$ 1,060,486
Weighted Average Coupon	3.40%	3.40%
Weighted Average Term to Maturity (Months)	351 months	354 months
Weighted Average LTV Ratio	62.00%	62.00%
Weighted Average FICO Scores	767	767
Settled Whole Loans | Agency eligible investor loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 1,026,214	$ 1,039,257
Settled Whole Loans | Agency eligible investor loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	7,595	21,229
Settled Whole Loans | Agency eligible investor loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	814	0
Settled Whole Loans | Agency eligible investor loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	192	0
Settled Whole Loans | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Deteriorated Loans	496,871	524,992
Unpaid Principal Balance (“UPB”)	$ 610,651	$ 643,187
Weighted Average Coupon	4.57%	4.55%
Weighted Average Term to Maturity (Months)	280 months	283 months
Weighted Average LTV Ratio	69.00%	69.00%
Settled Whole Loans | Purchased Credit Deteriorated Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 428,302	$ 456,924
Settled Whole Loans | Purchased Credit Deteriorated Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	51,517	50,048
Settled Whole Loans | Purchased Credit Deteriorated Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	18,942	18,736
Settled Whole Loans | Purchased Credit Deteriorated Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	111,890	117,479
Settled Whole Loans | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Deteriorated Loans	987,794	1,072,270
Unpaid Principal Balance (“UPB”)	$ 1,017,658	$ 1,073,544
Weighted Average Coupon	4.89%	4.87%
Weighted Average Term to Maturity (Months)	280 months	283 months
Weighted Average LTV Ratio	73.00%	73.00%
Settled Whole Loans | Purchased Non-Performing Loans | Current
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 464,770	$ 492,481
Settled Whole Loans | Purchased Non-Performing Loans | 30-59
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	85,856	87,041
Settled Whole Loans | Purchased Non-Performing Loans | 60-89
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	40,454	40,876
Settled Whole Loans | Purchased Non-Performing Loans | 90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	426,578	453,146
Settled Whole Loans | Residential whole loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Deteriorated Loans	8,232,996	7,913,000
Unpaid Principal Balance (“UPB”)	$ 8,497,405	$ 7,905,743
Weighted Average Coupon	4.96%	4.99%
Weighted Average Term to Maturity (Months)	298 months	301 months
Settled Whole Loans | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 160,500	$ 137,300
Weighted Average LTV Ratio	74.00%	71.00%