Residential Whole Loans (Narrative) (Details) - USD ($)	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2020	Dec. 31, 2019
Receivables [Abstract]
Total residential whole loans	$ 5,900,000,000		$ 5,900,000,000	$ 7,400,000,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loss on sale of financing receivable	(800,000)
Valuation allowance adjustment	70,181,000	$ (70,181,000)
Unpaid principal balance of loans	24,100,000		24,100,000
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	955,400,000		1,800,000,000
Realized loss on sale of financing receivable	(127,200,000)
Loss on sale of financing receivable			273,000,000.0
Valuation allowance adjustment	70,181,000	(70,181,000)
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	171,600,000		171,600,000	$ 99,900,000
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Valuation allowance adjustment	0	$ 0
Financing receivable, nonaccrual	122,000,000.0		122,000,000.0
Interest income			$ 0
Non-QM Loans Related to Reversed Valuation Allowance
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loss on sale of financing receivable	$ (57,000,000.0)