Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019	Sep. 30, 2018
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 64,226	$ 29,524	$ 171,725	$ 61,788
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	30,258	10,284	79,250	16,177
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	15,142	4,708	38,331	8,323
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	5,025	918	11,652	1,733
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,166	2,416	9,461	2,416
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	53,591	18,326	138,694	28,649
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 10,635	$ 11,198	$ 33,031	$ 33,139