Residential Whole Loans (Fair Value) (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Residential whole loans, at fair value	$ 1,381,583	$ 1,471,262
Excluded amount, unsettled purchases, residential whole loans, at fair value		194,700
Less than 60 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	666,026	610,290
Aggregate fair value	$ 641,616	$ 561,770
Weighted average LTV ratio	76.69%	76.18%
Number of loans | loan	3,159	2,898
60 Days to 89 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 58,160	$ 63,938
Aggregate fair value	$ 53,485	$ 54,947
Weighted average LTV ratio	79.48%	82.86%
Number of loans | loan	313	285
90 Days or More Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 767,320	$ 970,758
Aggregate fair value	$ 686,482	$ 854,545
Weighted average LTV ratio	89.69%	90.24%
Number of loans | loan	2,983	3,531
Residential whole loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Excluded amount, unsettled purchases, residential whole loans, at fair value		$ 194,700