Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2021	Dec. 31, 2020
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, total or weighted average	$ 5,226,424	$ 3,478,160
Unpaid Principal Balance (“UPB”)	$ 5,107,108	$ 3,467,099
Weighted Average Coupon	5.21%	6.04%
Weighted Average Term to Maturity (Months)	304 months	281 months
Settled Whole Loans | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 2,825,883	$ 2,336,117
Unpaid Principal Balance (“UPB”)	$ 2,737,998	$ 2,294,086
Weighted Average Coupon	5.36%	5.84%
Weighted Average Term to Maturity (Months)	350 months	351 months
Weighted Average LTV Ratio	64.00%	64.00%
Weighted Average FICO Scores	725	712
Settled Whole Loans | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 587,539	$ 563,430
Unpaid Principal Balance (“UPB”)	$ 594,366	$ 581,801
Weighted Average Coupon	7.27%	7.29%
Weighted Average Term to Maturity (Months)	8 months	3 months
Weighted Average LTV Ratio	66.00%	63.00%
Weighted Average FICO Scores	726	719
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 739,428	$ 442,456
Unpaid Principal Balance (“UPB”)	$ 717,552	$ 442,208
Weighted Average Coupon	5.69%	6.32%
Weighted Average Term to Maturity (Months)	330 months	324 months
Weighted Average LTV Ratio	70.00%	70.00%
Weighted Average FICO Scores	731	730
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 110,112	$ 136,157
Unpaid Principal Balance (“UPB”)	$ 120,444	$ 149,004
Weighted Average Coupon	2.86%	3.30%
Weighted Average Term to Maturity (Months)	164 months	171 months
Weighted Average LTV Ratio	38.00%	40.00%
Weighted Average FICO Scores	722	723
Settled Whole Loans | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased performing loans	$ 963,462
Unpaid Principal Balance (“UPB”)	$ 936,748
Weighted Average Coupon	3.40%
Weighted Average Term to Maturity (Months)	356 months
Weighted Average LTV Ratio	62.00%
Weighted Average FICO Scores	767
Settled Whole Loans | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value	$ 551,186	$ 630,339
Unpaid Principal Balance (“UPB”)	$ 674,367	$ 782,319
Weighted Average Coupon	4.55%	4.46%
Weighted Average Term to Maturity (Months)	284 months	287 months
Weighted Average LTV Ratio	69.00%	76.00%
Settled Whole Loans | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value	$ 1,140,837	$ 1,216,902
Unpaid Principal Balance (“UPB”)	$ 1,137,666	$ 1,282,093
Weighted Average Coupon	4.88%	4.87%
Weighted Average Term to Maturity (Months)	285 months	290 months
Weighted Average LTV Ratio	74.00%	80.00%
Settled Whole Loans | Residential whole loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value	$ 6,918,447	$ 5,325,401
Unpaid Principal Balance (“UPB”)	$ 6,919,141	$ 5,531,511
Weighted Average Coupon	5.10%	5.54%
Weighted Average Term to Maturity (Months)	299 months	284 months
Settled Whole Loans | Current | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 2,526,620	$ 2,042,405
Settled Whole Loans | Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	469,292	390,706
Settled Whole Loans | Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	690,822	411,377
Settled Whole Loans | Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	109,331	136,778
Settled Whole Loans | Current | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	933,633
Settled Whole Loans | Current | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	481,330	544,803
Settled Whole Loans | Current | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	517,924	497,299
Settled Whole Loans | 30-59 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	65,991	71,303
Settled Whole Loans | 30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	17,939	29,315
Settled Whole Loans | 30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,834	6,691
Settled Whole Loans | 30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,095	2,248
Settled Whole Loans | 30-59 | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	2,818
Settled Whole Loans | 30-59 | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	50,991	65,791
Settled Whole Loans | 30-59 | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	94,139	104,993
Settled Whole Loans | 60-89 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	18,902	35,697
Settled Whole Loans | 60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	3,432	25,433
Settled Whole Loans | 60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	1,033	3,907
Settled Whole Loans | 60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	616	1,155
Settled Whole Loans | 60-89 | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	297
Settled Whole Loans | 60-89 | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	18,857	26,697
Settled Whole Loans | 60-89 | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	39,605	54,180
Settled Whole Loans | 90 or more | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	126,485	144,681
Settled Whole Loans | 90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	103,703	136,347
Settled Whole Loans | 90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	23,863	20,233
Settled Whole Loans | 90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	9,402	8,823
Settled Whole Loans | 90 or more | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	0
Settled Whole Loans | 90 or more | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	123,189	145,028
Settled Whole Loans | 90 or more | Purchased Non-Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Aging by UPB	$ 485,998	$ 625,621