Summary of Significant Accounting Policies - MSR Related Assets, Residential Whole Loans, Cash and Goodwill (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($) segment	Dec. 31, 2016 USD ($)
Accounting Policies [Abstract]
Commitment to lend	$ 130,000
Amount drawn	$ 111,200
Number of reportable segments | segment	1
Number of operating segments | segment	1
Number of days considered to classify loans delinquent (less than)	60 days
LTV percentage (or less)	80.00%
Overnight money market funds	$ 354,000	$ 208,900
Restricted cash	13,307	58,463
Goodwill	$ 7,189	$ 7,189