Residential Whole Loans - Provision for Loan Losses (Details) - USD ($)	3 Months Ended		9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Sep. 30, 2017	Sep. 30, 2016
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 375,000	$ 1,023,000	$ 990,000	$ 1,165,000
Reversal of provisions for loan losses	(57,000)	0	(672,000)	(142,000)
Balance at the end of period	$ 318,000	$ 1,023,000	$ 318,000	$ 1,023,000