Fair Value of Financial Instruments - Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	$ 647,360	$ 623,276
Total Residential Whole Loans, Unpaid Principal Balance	810,263	786,330
Residential Whole Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	647,360	623,276
Total Residential Whole Loans, Unpaid Principal Balance	810,263	786,330
Total Residential Whole Loans, Difference	(162,903)	(163,054)
Residential Whole Loans 90 days or more past due, Fair Value	498,989	493,640
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance	639,747	637,459
Residential Whole Loans 90 days or more past due, Difference	$ (140,758)	$ (143,819)