Operating Group Information - Financial information (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Oct. 30, 2021	Oct. 31, 2020	Oct. 30, 2021	Oct. 31, 2020	Jan. 30, 2021
Operating groups
Net sales	$ 247,729	$ 175,135	$ 842,163	$ 527,466
Depreciation and amortization	9,877	10,572	29,252	34,223
Operating income (loss)	30,607	(13,718)	133,496	(107,224)
Interest expense, net	222	339	685	1,673
Earnings (loss) before income taxes	30,385	(14,057)	132,811	(108,897)
Assets	932,803	902,017	932,803	902,017	$ 865,634
Impairment of goodwill and intangible assets				60,452
Corporate and Other
Operating groups
Net sales	9,735	5,683	27,165	16,479
Depreciation and amortization	312	336	947	1,000
Operating income (loss)	6,053	1,192	(12,678)	(3,534)
Assets	193,812	52,993	193,812	52,993	76,705
Tommy Bahama | Operating Segments
Operating groups
Net sales	148,454	94,905	513,985	277,143
Depreciation and amortization	6,895	7,179	20,801	24,173
Operating income (loss)	5,531	(7,212)	73,515	(43,286)
Assets	531,534	616,049	531,534	616,049	569,854
Lilly Pulitzer | Operating Segments
Operating groups
Net sales	72,157	53,714	233,066	176,723
Depreciation and amortization	2,445	2,254	6,833	7,585
Operating income (loss)	15,985	5,266	61,713	25,676
Assets	173,104	182,020	173,104	182,020	176,467
Southern Tide | Operating Segments
Operating groups
Net sales	13,151	10,023	43,204	27,136
Depreciation and amortization	198	174	583	487
Operating income (loss)	2,690	(464)	8,893	(64,809)
Assets	31,092	30,172	31,092	30,172	31,641
Lanier Apparel | Operating Segments
Operating groups
Net sales	4,232	10,810	24,743	29,985
Depreciation and amortization	27	629	88	978
Operating income (loss)	348	(12,500)	2,053	(21,271)
Assets	$ 3,261	$ 20,783	$ 3,261	$ 20,783	$ 10,967