Operating Group Information - Financial information (Details) - USD ($) $ in Thousands	3 Months Ended			9 Months Ended
	Oct. 31, 2020	May 02, 2020	Nov. 02, 2019	Oct. 31, 2020	Nov. 02, 2019	Feb. 01, 2020
Operating groups
Net sales	$ 175,135		$ 241,221	$ 527,466	$ 825,194
Depreciation and amortization	10,572		10,193	34,223	30,179
Operating income (loss)	(13,718)		2,594	(107,224)	72,595
Interest expense, net	339		81	1,673	1,171
(Loss) earnings before income taxes	(14,057)		2,513	(108,897)	71,424
Total Assets	902,017		1,013,084	902,017	1,013,084	$ 1,033,369
Impairment of goodwill and intangible assets		$ 60,000		60,452
Corporate and Other
Operating groups
Net sales	5,683		4,679	16,479	13,680
Depreciation and amortization	336		323	1,000	1,024
Operating income (loss)	1,192		(3,152)	(3,534)	(13,380)
Total Assets	52,993		(406)	52,993	(406)	22,059
Tommy Bahama | Operating Segments
Operating groups
Net sales	94,905		127,023	277,143	480,623
Depreciation and amortization	7,179		7,073	24,173	20,820
Operating income (loss)	(7,212)		(7,739)	(43,286)	30,671
Total Assets	616,049		673,788	616,049	673,788	668,197
Lilly Pulitzer | Operating Segments
Operating groups
Net sales	53,714		71,659	176,723	219,809
Depreciation and amortization	2,254		2,554	7,585	7,618
Operating income (loss)	5,266		10,988	25,676	46,689
Total Assets	182,020		192,448	182,020	192,448	199,913
Lanier Apparel | Operating Segments
Operating groups
Net sales	10,810		28,758	29,985	75,378
Depreciation and amortization	629		108	978	313
Operating income (loss)	(12,500)		1,971	(21,271)	3,738
Total Assets	20,783		52,378	20,783	52,378	43,533
Southern Tide
Operating groups
Impairment of goodwill and intangible assets		$ 60,000
Southern Tide | Operating Segments
Operating groups
Net sales	10,023		9,102	27,136	35,704
Depreciation and amortization	174		135	487	404
Operating income (loss)	(464)		526	(64,809)	4,877
Total Assets	$ 30,172		$ 94,876	$ 30,172	$ 94,876	$ 99,667