Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($)	6 Months Ended	12 Months Ended
	Jun. 30, 2019	Dec. 31, 2018
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 3,646,329,000	$ 2,218,728,000
Purchased Credit Impaired Loans	$ 745,654,000	797,987,000
Weighted Average LTV Ratio	80.00%
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 2,290,713,000	1,354,774,000
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	859,705,000	494,576,000
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	295,461,000	145,327,000
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	200,450,000	$ 224,051,000
Certain Rehabilitation Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for losses	$ 450,000
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	88.80%	90.24%
Settled Whole Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value, total or weighted average	$ 4,392,433,000
Unpaid Principal Balance (“UPB”)	$ 4,522,887,000
Weighted Average Coupon	5.98%
Weighted Average Term to Maturity (Months)	272 months
Settled Whole Loans | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 2,290,713,000
Unpaid Principal Balance (“UPB”)	$ 2,217,845,000
Weighted Average Coupon	6.18%
Weighted Average Term to Maturity (Months)	364 months
Weighted Average LTV Ratio	66.00%
Settled Whole Loans | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 860,155,000
Unpaid Principal Balance (“UPB”)	$ 860,155,000
Weighted Average Coupon	7.33%
Weighted Average Term to Maturity (Months)	9 months
Weighted Average LTV Ratio	65.00%
Settled Whole Loans | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 295,461,000
Unpaid Principal Balance (“UPB”)	$ 294,110,000
Weighted Average Coupon	6.21%
Weighted Average Term to Maturity (Months)	327 months
Weighted Average LTV Ratio	69.00%
Settled Whole Loans | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 200,450,000
Unpaid Principal Balance (“UPB”)	$ 217,635,000
Weighted Average Coupon	4.43%
Weighted Average Term to Maturity (Months)	186 months
Weighted Average LTV Ratio	47.00%
Settled Whole Loans | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Credit Impaired Loans	$ 745,654,000
Unpaid Principal Balance (“UPB”)	$ 933,142,000
Weighted Average Coupon	4.43%
Weighted Average Term to Maturity (Months)	298 months
Weighted Average LTV Ratio	85.00%
Settled Whole Loans | Certain Rehabilitation Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans	$ 215,500,000
Weighted Average LTV Ratio	68.00%
Settled Whole Loans | Current | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 2,177,570,000
Settled Whole Loans | Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	775,479,000
Settled Whole Loans | Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	289,768,000
Settled Whole Loans | Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	213,229,000
Settled Whole Loans | 30-59 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	21,827,000
Settled Whole Loans | 30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	45,955,000
Settled Whole Loans | 30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	3,219,000
Settled Whole Loans | 30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	3,071,000
Settled Whole Loans | 60-89 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	10,622,000
Settled Whole Loans | 60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	16,351,000
Settled Whole Loans | 60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	1,123,000
Settled Whole Loans | 60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	657,000
Settled Whole Loans | 90 or more | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	7,826,000
Settled Whole Loans | 90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	22,370,000
Settled Whole Loans | 90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	0
Settled Whole Loans | 90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	678,000
Settled Whole Loans | 90 or more | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 103,029,000