Other Assets (Real Estate Owned - Activity) (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019 USD ($) property	Jun. 30, 2018 USD ($) property	Jun. 30, 2019 USD ($) property	Jun. 30, 2018 USD ($) property	Dec. 31, 2018 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 290,587	$ 182,940	$ 249,413	$ 152,356
Adjustments to record at lower of cost or fair value	(1,315)	(4,121)	(5,388)	(7,536)
Transfer from residential whole loans	66,483	48,699	131,644	103,521
Purchases and capital improvements, net	5,274	2,604	11,197	5,282
Disposals	(26,960)	(37,960)	(52,797)	(61,461)
Balance at end of period	$ 334,069	$ 192,162	$ 334,069	$ 192,162
Number of properties | property	1,362	884	1,362	884	1,093
Gain recorded on transfer from residential whole loans to real estate owned	$ 6,500	$ 5,300	$ 11,300	$ 11,700
Properties sold during period | property	152	212	289	380
Proceeds from sale of real estate	$ 29,200	$ 40,600	$ 57,000	$ 66,100
Gain on sales of real estate owned	$ 2,300	$ 2,700	$ 3,692	$ 4,673