Operating Group Information - Financial information (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Aug. 01, 2020	Aug. 03, 2019	Aug. 01, 2020	Aug. 03, 2019	Feb. 01, 2020
Operating groups
Net sales	$ 191,988	$ 302,000	$ 352,331	$ 583,973
Depreciation and amortization	12,958	9,849	23,651	19,986
Operating income (loss)	(8,017)	40,259	(93,506)	70,001
Interest expense, net	676	419	1,334	1,090
(Loss) earnings before income taxes	(8,693)	39,840	(94,840)	68,911
Total Assets	952,549	1,010,194	952,549	1,010,194	$ 1,033,369
Impairment of goodwill and intangible assets			60,452
Corporate and Other
Operating groups
Net sales	5,612	4,641	10,796	9,001
Depreciation and amortization	333	285	664	625
Operating income (loss)	(4,456)	(5,642)	(4,726)	(10,228)
Total Assets	91,344	4,656	91,344	4,656	22,059
Tommy Bahama | Operating Segments
Operating groups
Net sales	95,254	188,870	182,238	353,600
Depreciation and amortization	9,194	6,907	16,994	13,747
Operating income (loss)	(12,712)	23,218	(36,074)	38,410
Total Assets	605,448	655,836	605,448	655,836	668,197
Lilly Pulitzer | Operating Segments
Operating groups
Net sales	73,860	75,555	123,009	148,150
Depreciation and amortization	3,015	2,381	5,331	5,064
Operating income (loss)	16,264	20,449	20,410	35,701
Total Assets	191,099	208,121	191,099	208,121	199,913
Lanier Apparel | Operating Segments
Operating groups
Net sales	8,450	20,466	19,175	46,620
Depreciation and amortization	247	141	349	281
Operating income (loss)	(6,134)	400	(8,771)	1,767
Total Assets	31,740	43,545	31,740	43,545	43,533
Southern Tide
Operating groups
Impairment of goodwill and intangible assets			60,000
Southern Tide | Operating Segments
Operating groups
Net sales	8,812	12,468	17,113	26,602
Depreciation and amortization	169	135	313	269
Operating income (loss)	(979)	1,834	(64,345)	4,351
Total Assets	$ 32,918	$ 98,036	$ 32,918	$ 98,036	$ 99,667