Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($) $ in Thousands		3 Months Ended		12 Months Ended
		Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 5,189,634		$ 5,370,653
Purchased Credit Deteriorated Loans		744,408		698,717
Residential whole loans, at carrying value, total or weighted average		$ 4,968,583
Weighted Average LTV Ratio		80.00%
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 3,538,725		3,707,245
Residential whole loans, at carrying value, total or weighted average		2,573,266
Loans net carrying amount	[1],[2]	895,300
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		978,965		1,026,097
Residential whole loans, at carrying value, total or weighted average		978,965
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		506,352		460,742
Residential whole loans, at carrying value, total or weighted average		506,352
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		165,592		176,569
Residential whole loans, at carrying value, total or weighted average		$ 165,592
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio		88.12%	89.69%
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value, total or weighted average		$ 5,716,031		6,066,345
Amortized Cost Basis		5,934,042		6,069,370
Unpaid Principal Balance (“UPB”)		$ 5,944,079		$ 6,141,421
Weighted Average Coupon		5.88%		5.96%
Weighted Average Term to Maturity (Months)		285 months		288 months
Settled Whole Loans | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 3,434,894		$ 3,706,857
Amortized Cost Basis		3,538,725		3,707,245
Unpaid Principal Balance (“UPB”)		$ 3,424,646		$ 3,592,701
Weighted Average Coupon		5.84%		5.96%
Weighted Average Term to Maturity (Months)		363 months		368 months
Weighted Average LTV Ratio		66.00%		67.00%
Weighted Average FICO Scores		717		716
Settled Whole Loans | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 943,332		$ 1,023,766
Amortized Cost Basis		978,965		1,026,097
Unpaid Principal Balance (“UPB”)		$ 978,965		$ 1,026,097
Weighted Average Coupon		7.24%		7.30%
Weighted Average Term to Maturity (Months)		7 months		8 months
Weighted Average LTV Ratio		64.00%		64.00%
Weighted Average FICO Scores		720		717
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 498,921		$ 460,679
Amortized Cost Basis		506,352		460,741
Unpaid Principal Balance (“UPB”)		$ 501,925		$ 457,146
Weighted Average Coupon		6.28%		6.29%
Weighted Average Term to Maturity (Months)		322 months		324 months
Weighted Average LTV Ratio		70.00%		70.00%
Weighted Average FICO Scores		734		734
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 165,343		$ 176,569
Amortized Cost Basis		165,592		176,569
Unpaid Principal Balance (“UPB”)		$ 180,421		$ 192,151
Weighted Average Coupon		4.11%		4.24%
Weighted Average Term to Maturity (Months)		178 months		181 months
Weighted Average LTV Ratio		42.00%		46.00%
Weighted Average FICO Scores		723		723
Settled Whole Loans | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Deteriorated Loans		$ 673,541		$ 698,474
Amortized Cost Basis		744,408		698,718
Unpaid Principal Balance (“UPB”)		$ 858,122		$ 873,326
Weighted Average Coupon		4.46%		4.46%
Weighted Average Term to Maturity (Months)		292 months		294 months
Weighted Average LTV Ratio		80.00%		81.00%
Settled Whole Loans | Certain Rehabilitation Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 259,400		$ 269,200
Weighted Average LTV Ratio		68.00%		69.00%
Settled Whole Loans | Current | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		$ 3,450,648		$ 3,492,533
Settled Whole Loans | Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		806,413		868,281
Settled Whole Loans | Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		482,499		432,936
Settled Whole Loans | Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		160,944		187,683
Settled Whole Loans | 30-59 | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		50,584		59,963
Settled Whole Loans | 30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		61,723		67,747
Settled Whole Loans | 30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		17,536		15,948
Settled Whole Loans | 30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		1,670		2,164
Settled Whole Loans | 60-89 | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		13,058		19,605
Settled Whole Loans | 60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		20,973		27,437
Settled Whole Loans | 60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		2,009		2,047
Settled Whole Loans | 60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		1,099		430
Settled Whole Loans | 90 or more | Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		24,435		20,600
Settled Whole Loans | 90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		89,856		62,632
Settled Whole Loans | 90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		4,308		6,215
Settled Whole Loans | 90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		1,879		1,874
Settled Whole Loans | 90 or more | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)		$ 87,179		$ 108,998

[1] Includes Non-QM loans held-for-sale with an amortized cost of $965.5 million and a net carrying value of $895.3 million at March 31, 2020.

[2] Includes approximately $185.9 million and $186.4 million of Residential whole loans, at carrying value and $516.4 million and $567.4 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at March 31, 2020 and December 31, 2019, respectively. Such assets can be used only to settle the obligations of each respective VIE