Residential Whole Loans (Fair Value) (Details) $ in Thousands	9 Months Ended
	Sep. 30, 2020 USD ($) loan	Sep. 30, 2019 loan	Dec. 31, 2019 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Total Residential whole loans, at fair value	$ 1,229,664		$ 1,381,583
Less than 60 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	599,461		666,026
Aggregate fair value	$ 577,761		641,616
Weighted Average LTV Ratio	74.33%	76.69%
Number of loans | loan	3,038	3,159
60 Days to 89 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 55,183		58,160
Aggregate fair value	$ 49,188		53,485
Weighted Average LTV Ratio	83.62%	79.48%
Number of loans | loan	259	313
90 Days or More Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 679,211		767,320
Aggregate fair value	$ 602,715		$ 686,482
Weighted Average LTV Ratio	87.82%	89.69%
Number of loans | loan	2,532	2,983