Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 45,340	$ 83,486
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	22,114	49,070
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	6,668	15,327
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	6,278	7,343
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,990	2,600
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	37,050	74,340
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 8,290	$ 9,146