Residential Whole Loans (Provision for Loan Losses) (Details) - Purchased credit impaired loans - USD ($) $ in Thousands	3 Months Ended
Mar. 31, 2019	Mar. 31, 2018
Allowance for Loan and Lease Losses [Roll Forward]
Balance at the beginning of period	$ 968	$ 330
Provisions for loan losses	183	50
Balance at the end of period	$ 1,151	$ 380

Amount of expense related to estimated loss from loan and lease transactions.

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 230
-SubTopic 10
-Section 45
-Paragraph 28
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=109223946&loc=d3e3602-108585

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 942
-SubTopic 220
-Section S99
-Paragraph 1
-Subparagraph (SX 210.9-04.11)
-URI http://asc.fasb.org/extlink&oid=114873765&loc=SL114874048-224260

Reference 3: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 11B
-Subparagraph (c)(2)
-URI http://asc.fasb.org/extlink&oid=84173941&loc=SL6953423-111524

Name:	us-gaap_ProvisionForLoanAndLeaseLosses
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	duration