Residential Whole Loans (Narrative) (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021 USD ($) loan	Jun. 30, 2020 USD ($)	Jun. 30, 2021 USD ($) loan	Jun. 30, 2020 USD ($)	Dec. 31, 2020 USD ($)
Receivables [Abstract]
Total residential whole loans	$ 5,600,000		$ 5,600,000		$ 5,300,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of whole loans sold | loan	0		0
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale		$ 24,100
Loss on sale of financing receivable		800
Financing receivable, nonaccrual	$ 660,700		$ 660,700		730,900
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale		955,400		$ 1,800,000
Loss on sale of financing receivable		$ 127,200		$ 273,000
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	307,300		307,300		373,300
Purchased credit deteriorated loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	122,200		122,200		151,400
Without Associated Credit Losses
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, nonaccrual	$ 117,300		$ 117,300		$ 130,700