Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2019	Mar. 31, 2018
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 49,620	$ 14,329
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	22,414	1,708
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	9,933	1,345
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	2,701	245
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,173	0
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	38,221	3,298
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 11,399	$ 11,031