Residential Whole Loans (Purchased Performing Loans) (Details)	3 Months Ended
	Mar. 31, 2019 USD ($) loan	Mar. 31, 2018 USD ($)
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of other loans at carrying value | loan	56
Unpaid balance of other loans	$ 26,300,000
Total outstanding principal balance of all of the Company’s purchased performing loans	0.90%
Allowance for loan loss	$ 500,000
Provisions for loan losses	622,000	$ 0
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Commitment to lend, unfunded	$ 53,500,000