Other Assets - Real Estate Owned - Activity (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 28,026	$ 5,492
Adjustments to record at lower of cost or fair value	(7,527)	(3,475)
Transfer from residential whole loans	91,896	30,104	$ 2,904
Purchases and capital improvements	2,825	2,461
Disposals	(34,717)	(6,556)
Balance at end of period	80,503	28,026	$ 5,492
Gain recorded on transfer from residential whole loans to real estate owned	$ 2,900	$ 1,700