Residential Whole Loans (Fair Value) (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2020 USD ($) loan	Jun. 30, 2019 loan	Dec. 31, 2019 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Total Residential whole loans, at fair value	$ 1,200,981		$ 1,381,583
Less than 60 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	593,389		666,026
Aggregate fair value	$ 545,953		641,616
Weighted Average LTV Ratio	74.64%	76.69%
Number of loans | loan	2,981	3,159
60 Days to 89 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 79,684		58,160
Aggregate fair value	$ 69,303		53,485
Weighted Average LTV Ratio	82.43%	79.48%
Number of loans | loan	342	313
90 Days or More Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 694,590		767,320
Aggregate fair value	$ 585,725		$ 686,482
Weighted Average LTV Ratio	88.06%	89.69%
Number of loans | loan	2,642	2,983