Residential Whole Loans (Residential Whole Loans, at Carrying Value - Additional Information) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans held at carrying value excluded	$ 211,100
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	797,987	$ 790,879
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	1,354,774	55,612
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	494,576	56,706
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	145,327	5,319
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	224,051	$ 0
Other residential whole loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans held at carrying value excluded	$ 16,400
90 or more
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	90.24%	94.50%
Settled Whole Loans [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Residential whole loans, at carrying value, total or weighted average	$ 3,000,301
Unpaid Principal Balance (“UPB”)	$ 3,178,153
Weighted Average Coupon	5.68%
Weighted Average Term to Maturity (Months)	275 months
Settled Whole Loans [Member] | Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 797,987
Unpaid Principal Balance (“UPB”)	$ 1,002,261
Weighted Average Coupon	4.38%
Weighted Average Term to Maturity (Months)	302 months
Weighted Average LTV Ratio	86.00%
Settled Whole Loans [Member] | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 1,338,360
Unpaid Principal Balance (“UPB”)	$ 1,293,805
Weighted Average Coupon	6.19%
Weighted Average Term to Maturity (Months)	361 months
Weighted Average LTV Ratio	66.00%
Settled Whole Loans [Member] | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 494,576
Unpaid Principal Balance (“UPB”)	$ 494,576
Weighted Average Coupon	7.39%
Weighted Average Term to Maturity (Months)	9 months
Weighted Average LTV Ratio	65.00%
Settled Whole Loans [Member] | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 145,327
Unpaid Principal Balance (“UPB”)	$ 144,972
Weighted Average Coupon	5.92%
Weighted Average Term to Maturity (Months)	355 months
Weighted Average LTV Ratio	70.00%
Settled Whole Loans [Member] | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 224,051
Unpaid Principal Balance (“UPB”)	$ 242,539
Weighted Average Coupon	4.25%
Weighted Average Term to Maturity (Months)	191 months
Weighted Average LTV Ratio	47.00%
Settled Whole Loans [Member] | Certain rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans	$ 79,000
Weighted Average LTV Ratio	67.00%
Settled Whole Loans [Member] | Current | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 1,271,410
Settled Whole Loans [Member] | Current | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	465,033
Settled Whole Loans [Member] | Current | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	143,226
Settled Whole Loans [Member] | Current | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	233,105
Settled Whole Loans [Member] | 30-59 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	14,535
Settled Whole Loans [Member] | 30-59 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	18,353
Settled Whole Loans [Member] | 30-59 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	1,225
Settled Whole Loans [Member] | 30-59 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	6,439
Settled Whole Loans [Member] | 60-89 | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	4,802
Settled Whole Loans [Member] | 60-89 | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	6,451
Settled Whole Loans [Member] | 60-89 | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	0
Settled Whole Loans [Member] | 60-89 | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	1,618
Settled Whole Loans [Member] | 90 or more | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	3,058
Settled Whole Loans [Member] | 90 or more | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	4,739
Settled Whole Loans [Member] | 90 or more | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	521
Settled Whole Loans [Member] | 90 or more | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unpaid Principal Balance (“UPB”)	$ 1,377