Distressed and Other Residential Mortgage Loans, Net - Allowance for Loan Losses (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Balance at beginning of period	$ 3,759	$ 4,191	$ 3,782
Provisions for loan losses	25	166	475
Transfer to real estate owned	(167)	0	(6)
Charge-offs	(109)	(598)	(60)
Balance at the end of period	$ 3,508	$ 3,759	$ 4,191