Other Assets (Real Estate Owned - Activity) (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2019 USD ($) property	Mar. 31, 2018 USD ($) property	Dec. 31, 2018 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 249,413	$ 152,356
Adjustments to record at lower of cost or fair value	(4,072)	(3,415)
Transfer from residential whole loans	65,160	54,822
Purchases and capital improvements	5,923	2,678
Disposals	(25,837)	(23,501)
Balance at end of period	$ 290,587	$ 182,940
Number of properties | property	1,233	845	1,093
Gain recorded on transfer from residential whole loans to real estate owned	$ 4,600	$ 6,400
Properties sold during period | property	137	168
Proceeds from sale of real estate	$ 27,800	$ 25,500
Gain on sales of real estate owned	$ 1,398	$ 1,993