Other Assets (Real Estate Owned - Activity) (Details) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020 USD ($) property	Sep. 30, 2019 USD ($) property	Sep. 30, 2020 USD ($) property	Sep. 30, 2019 USD ($) property	Dec. 31, 2019 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 348,516	$ 334,069	$ 411,659	$ 249,413
Adjustments to record at lower of cost or fair value	93	(3,875)	(11,796)	(9,264)
Transfer from residential whole loans	15,672	61,888	74,891	193,531
Purchases and capital improvements, net	536	5,108	9,334	16,307
Disposals	(65,951)	(20,990)	(185,222)	(73,787)
Balance at end of period	$ 298,866	$ 376,200	$ 298,866	$ 376,200
Number of properties | property	1,131	1,508	1,131	1,508	1,652
Gain recorded on transfer from residential whole loans to real estate owned	$ 834	$ 5,000	$ 4,100	$ 16,100
Properties sold during period | property	267	142	812	431
Proceeds from sale of real estate	$ 69,900	$ 23,000	$ 195,200	$ 80,000
Gain on sales of real estate owned	$ 3,900	$ 2,100	$ 10,000	$ 5,800