Residential Whole Loans (Residential Whole Loans, at Carrying Value) (Details) $ in Thousands		Jun. 30, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 3,646,329	$ 2,218,728
Purchased Credit Impaired Loans		745,654	797,987
Total Residential whole loans, at carrying value	[1]	$ 4,391,983	$ 3,016,715
Number of loans | loan		14,091	11,149
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 2,290,713	$ 1,354,774
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		859,705	494,576
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		295,461	145,327
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased Performing Loans		$ 200,450	$ 224,051

[1] Includes approximately $197.0 million and $209.4 million of Residential whole loans, at carrying value and $616.0 million and $694.7 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at June 30, 2019 and December 31, 2018, respectively. Such assets can be used only to settle the obligations of each respective VIE.