Residential Whole Loans (Fair Value) (Details) $ in Thousands		9 Months Ended	12 Months Ended
		Sep. 30, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio		80.00%
Total Residential whole loans, at fair value	[1]	$ 1,453,169	$ 1,471,262
Residential loans excluded, not yet closed		59,500	194,700
Less than 60 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance		654,542	610,290
Aggregate fair value		$ 617,172	$ 561,770
Weighted Average LTV Ratio		77.08%	76.18%
Number of loans | loan		3,140	2,898
60 Days to 89 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance		$ 66,657	$ 63,938
Aggregate fair value		$ 60,262	$ 54,947
Weighted Average LTV Ratio		77.02%	82.86%
Number of loans | loan		340	285
90 Days or More Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance		$ 871,647	$ 970,758
Aggregate fair value		$ 775,735	$ 854,545
Weighted Average LTV Ratio		78.26%	90.24%
Number of loans | loan		3,397	3,531

[1] Includes approximately $189.9 million and $209.4 million of Residential whole loans, at carrying value and $593.3 million and $694.7 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at September 30, 2019 and December 31, 2018, respectively. Such assets can be used only to settle the obligations of each respective VIE.