Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2018	Dec. 31, 2017
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value, total or weighted average	$ 2,464,794
Unpaid Principal Balance (“UPB”)	$ 2,660,686
Weighted Average Coupon	5.46%
Weighted Average Term to Maturity (Months)	278 months
Weighted Average LTV Ratio	80.00%
Other loans held at carrying value excluded	$ 8,600
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	825,614	$ 790,879
Unpaid Principal Balance (“UPB”)	$ 1,034,890
Weighted Average Coupon	4.36%
Weighted Average Term to Maturity (Months)	304 months
Weighted Average LTV Ratio	87.00%
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value	$ 989,818	55,612
Unpaid Principal Balance (“UPB”)	$ 956,162
Weighted Average Coupon	6.18%
Weighted Average Term to Maturity (Months)	358 months
Weighted Average LTV Ratio	66.00%
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value	$ 329,301	56,706
Unpaid Principal Balance (“UPB”)	$ 329,301
Weighted Average Coupon	7.51%
Weighted Average Term to Maturity (Months)	9 months
Weighted Average LTV Ratio	65.00%
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value	$ 79,699	5,319
Unpaid Principal Balance (“UPB”)	$ 79,563
Weighted Average Coupon	5.84%
Weighted Average Term to Maturity (Months)	355 months
Weighted Average LTV Ratio	68.00%
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value	$ 247,135	$ 0
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value	240,362
Unpaid Principal Balance (“UPB”)	$ 260,770
Weighted Average Coupon	4.05%
Weighted Average Term to Maturity (Months)	194 months
Weighted Average LTV Ratio	49.00%
Other residential whole loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans held at carrying value excluded	$ 6,800
Current | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	944,564
Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	309,951
Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	78,172
Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	239,025
30-59 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	8,564
30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	12,836
30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	870
30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	20,373
60-89 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	1,865
60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	3,620
60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	0
60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 635
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	89.48%	94.50%
90 or more | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 1,169
90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	2,894
90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	521
90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 737