Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($) $ in Thousands	6 Months Ended		12 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Dec. 31, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 4,086,606		$ 5,370,653
Purchased Credit Deteriorated Loans	726,513		698,717
Residential whole loans, at carrying value, total or weighted average	$ 4,813,119
Weighted Average LTV Ratio	80.00%
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 2,574,184		3,707,245
Residential whole loans, at carrying value, total or weighted average	2,574,184
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	862,895		1,026,097
Residential whole loans, at carrying value, total or weighted average	862,895
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	494,248		460,742
Residential whole loans, at carrying value, total or weighted average	494,248
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	155,279		176,569
Residential whole loans, at carrying value, total or weighted average	$ 155,279
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	88.06%	89.69%
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value, total or weighted average	$ 4,676,530		6,066,345
Amortized Cost Basis	4,813,119		6,069,370
Unpaid Principal Balance (“UPB”)	$ 4,862,531		$ 6,141,421
Weighted Average Coupon	5.86%		5.96%
Weighted Average Term to Maturity (Months)	272 months		288 months
Settled Whole Loans | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 2,542,831		$ 3,706,857
Amortized Cost Basis	2,574,184		3,707,245
Unpaid Principal Balance (“UPB”)	$ 2,501,547		$ 3,592,701
Weighted Average Coupon	5.87%		5.96%
Weighted Average Term to Maturity (Months)	354 months		368 months
Weighted Average LTV Ratio	64.00%		67.00%
Weighted Average FICO Scores	712		716
Settled Whole Loans | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 832,895		$ 1,023,766
Amortized Cost Basis	862,895		1,026,097
Unpaid Principal Balance (“UPB”)	$ 862,895		$ 1,026,097
Weighted Average Coupon	7.26%		7.30%
Weighted Average Term to Maturity (Months)	6 months		8 months
Weighted Average LTV Ratio	63.00%		64.00%
Weighted Average FICO Scores	720		717
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 487,317		$ 460,679
Amortized Cost Basis	494,248		460,741
Unpaid Principal Balance (“UPB”)	$ 489,947		$ 457,146
Weighted Average Coupon	6.28%		6.29%
Weighted Average Term to Maturity (Months)	321 months		324 months
Weighted Average LTV Ratio	70.00%		70.00%
Weighted Average FICO Scores	734		734
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 155,055		$ 176,569
Amortized Cost Basis	155,279		176,569
Unpaid Principal Balance (“UPB”)	$ 169,469		$ 192,151
Weighted Average Coupon	3.76%		4.24%
Weighted Average Term to Maturity (Months)	176 months		181 months
Weighted Average LTV Ratio	42.00%		46.00%
Weighted Average FICO Scores	723		723
Settled Whole Loans | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Deteriorated Loans	$ 658,432		$ 698,474
Amortized Cost Basis	726,513		698,718
Unpaid Principal Balance (“UPB”)	$ 838,673		$ 873,326
Weighted Average Coupon	4.46%		4.46%
Weighted Average Term to Maturity (Months)	291 months		294 months
Weighted Average LTV Ratio	79.00%		81.00%
Settled Whole Loans | Certain Rehabilitation Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 280,600		$ 269,200
Weighted Average LTV Ratio	68.00%		69.00%
Settled Whole Loans | Current | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	$ 2,502,521		$ 3,492,533
Settled Whole Loans | Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	620,315		868,281
Settled Whole Loans | Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	444,308		432,936
Settled Whole Loans | Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	150,800		187,683
Settled Whole Loans | 30-59 | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	24,927		59,963
Settled Whole Loans | 30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	60,762		67,747
Settled Whole Loans | 30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	25,428		15,948
Settled Whole Loans | 30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	1,740		2,164
Settled Whole Loans | 60-89 | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	23,192		19,605
Settled Whole Loans | 60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	65,226		27,437
Settled Whole Loans | 60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	12,730		2,047
Settled Whole Loans | 60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	442		430
Settled Whole Loans | 90 or more | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	23,544		20,600
Settled Whole Loans | 90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	116,592		62,632
Settled Whole Loans | 90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	11,782		6,215
Settled Whole Loans | 90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	2,297		1,874
Settled Whole Loans | 90 or more | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Amortized Cost Basis	$ 105,536		$ 108,998