Residential Whole Loans (Components of Interest Income) (Details) - Residential whole loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 243,980	$ 100,921	$ 36,187
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	116,282	31,036	84
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	54,419	15,975	431
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	17,742	3,315	15
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	12,191	5,818	0
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	200,634	56,144	530
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 43,346	$ 44,777	$ 35,657