Residential Whole Loans (Fair Value) (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2021 USD ($) loan	Mar. 31, 2020 loan	Dec. 31, 2020 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio	80.00%
Total Residential whole loans, at fair value	$ 1,207,997		$ 1,216,902
Unsettled residential whole loans	112,200
Less than 60 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	590,813		602,292
Aggregate fair value	$ 596,805		595,521
Weighted Average LTV Ratio	69.46%	72.57%
Number of loans | loan	2,975	3,033
60 Days to 89 Days Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 58,625		54,180
Aggregate fair value	$ 56,021		49,652
Weighted Average LTV Ratio	70.56%	82.11%
Number of loans | loan	293	263
90 Days or More Past Due:
Accounts, Notes, Loans and Financing Receivable [Line Items]
Outstanding principal balance	$ 584,025		625,621
Aggregate fair value	$ 555,171		$ 571,729
Weighted Average LTV Ratio	82.56%	86.78%
Number of loans | loan	2,170	2,326