Real Estate Owned - Activity (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 43,291	$ 8,333	$ 28,026	$ 5,492
Adjustments to record at lower of cost or fair value	(1,219)	(1,379)	(2,996)	(1,379)
Transfer from residential whole loans	22,711	3,554	44,991	7,002
Purchases and capital improvements	1,379	0	1,789	0
Disposals	(9,378)	(1,651)	(15,026)	(2,258)
Balance at end of period	56,784	8,857	56,784	8,857
Gain recorded on transfer from residential whole loans to real estate owned	$ 1,300	$ 309	$ 1,700	$ 594