Residential Whole Loans (Residential Whole Loans, at Carrying Value) (Details) $ in Thousands		Jun. 30, 2018 USD ($) loan	Dec. 31, 2017 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total Residential whole loans, at carrying value	[1]	$ 1,906,242	$ 908,516
Number of loans | loan		8,300	4,800
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchased credit impaired loans		$ 804,848	$ 790,879
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value		626,927	55,612
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value		162,741	56,706
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value		55,571	5,319
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value		256,155	0
Total other loans at carrying value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Other loans at carrying value		$ 1,101,394	$ 117,637

[1] Includes approximately $199.8 million and $183.2 million of Residential whole loans, at carrying value and $476.2 million and $289.3 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at June 30, 2018 and December 31, 2017, respectively. Such assets can be used only to settle the obligations of the VIEs.