Fair Value of Financial Instruments - Difference Between Fair Value and Unpaid Principal Balance (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2014	Dec. 31, 2013
Fair Value, Option, Quantitative Disclosures [Line Items]
Total loans, Fair Value	$ 143,472	$ 0
Total loans, Unpaid Principal Balance	182,600
Residential whole loans
Fair Value, Option, Quantitative Disclosures [Line Items]
Total loans, Fair Value	143,472
Total loans, Unpaid Principal Balance	182,613
Total loans, Difference	(39,141)
Loans 90 days or more past due, Fair Value	128,591
Loans 90 days or more past due, Unpaid Principal Balance	165,358
Loans 90 days or more past due, Difference	$ (36,767)