Residential Whole Loans (Provision for Loan Losses) (Details) - Purchased credit impaired loans - USD ($) $ in Thousands	3 Months Ended	6 Months Ended
Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 380	$ 769	$ 330	$ 990
Reversal of provisions for loan losses	(83)	(394)	(33)	(615)
Balance at the end of period	$ 297	$ 375	$ 297	$ 375

Amount of expense related to estimated loss from loan and lease transactions.

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 230
-SubTopic 10
-Section 45
-Paragraph 28
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=109223946&loc=d3e3602-108585

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 942
-SubTopic 220
-Section S99
-Paragraph 1
-Subparagraph (SX 210.9-04.11)
-URI http://asc.fasb.org/extlink&oid=114873765&loc=SL114874048-224260

Reference 3: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 11B
-Subparagraph (c)(2)
-URI http://asc.fasb.org/extlink&oid=84173941&loc=SL6953423-111524

Name:	us-gaap_ProvisionForLoanAndLeaseLosses
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	duration