Residential Whole Loans - Provision for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
Mar. 31, 2017	Mar. 31, 2016
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 990	$ 1,165
Reversal of provisions for loan losses	(221)	(37)
Balance at the end of period	$ 769	$ 1,128

Amount of expense related to estimated loss from loan and lease transactions.

Reference 1: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 942
-SubTopic 225
-Section S99
-Paragraph 1
-Subparagraph (SX 210.9-04.11)
-URI http://asc.fasb.org/extlink&oid=6879574&loc=d3e536633-122882

Reference 2: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 230
-SubTopic 10
-Section 45
-Paragraph 28
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=66902209&loc=d3e3602-108585

Reference 3: http://www.xbrl.org/2003/role/presentationRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 11B
-Subparagraph (c)(2)
-URI http://asc.fasb.org/extlink&oid=66008989&loc=SL6953423-111524

Name:	us-gaap_ProvisionForLoanAndLeaseLosses
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	duration