Residential Whole Loans (Narrative) (Details)	3 Months Ended
	Mar. 31, 2021 USD ($) loan	Mar. 31, 2020 USD ($)	Dec. 31, 2020 USD ($)
Receivables [Abstract]
Total residential whole loans	$ 5,200,000,000		$ 5,300,000,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of whole loans sold | loan	0
Interest income	$ 0	$ 0
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	659,900,000
Loss on sale of financing receivable	145,800,000
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	363,300,000		373,300,000
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	146,400,000		151,400,000
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 132,300,000		$ 130,700,000