Other Assets - Real Estate Owned - Activity (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 80,503	$ 28,026
Adjustments to record at lower of cost or fair value	(11,018)	(7,527)
Transfer from residential whole loans	136,734	91,896	$ 30,104
Purchases and capital improvements	19,801	2,825
Disposals	(73,664)	(34,717)
Balance at end of period	152,356	80,503	$ 28,026
Gain recorded on transfer from residential whole loans to real estate owned	$ 10,200	$ 2,900