Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 54,393	$ 64,226	$ 207,306	$ 171,725
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	25,884	30,258	112,212	79,250
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	10,863	15,142	39,502	38,331
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	6,917	5,025	21,528	11,652
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,945	3,166	6,799	9,461
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	45,609	53,591	180,041	138,694
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 8,784	$ 10,635	$ 27,265	$ 33,031