Residential Whole Loans (Narrative) (Details) - USD ($)	3 Months Ended			9 Months Ended
	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2020	Dec. 31, 2019
Receivables [Abstract]
Total residential whole loans	$ 5,600,000,000			$ 5,600,000,000	$ 7,400,000,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loss on sale of financing receivable	(800,000)
Valuation allowance adjustment		$ 70,181,000	$ (70,181,000)
Interest income				0
Unpaid principal balance of loans	24,100,000			24,100,000
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale				1,800,000,000
Loss on sale of financing receivable				273,000,000.0
Valuation allowance adjustment		70,181,000	(70,181,000)
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	345,600,000			345,600,000	$ 99,900,000
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Valuation allowance adjustment		$ 0	$ 0
Financing receivable, nonaccrual	148,700,000			148,700,000
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 134,800,000			$ 134,800,000