Other Assets - Real Estate Owned - Activity (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2017	Mar. 31, 2016
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 80,503	$ 28,026
Adjustments to record at lower of cost or fair value	(1,823)	(1,777)
Transfer from residential whole loans	31,098	22,280
Purchases and capital improvements	774	410
Disposals	(11,844)	(5,648)
Balance at end of period	98,708	43,291
Gain recorded on transfer from residential whole loans to real estate owned	$ 1,300	$ 441