Operating Group Information - Sales by operating group (Details) - USD ($) $ in Thousands	12 Months Ended
	Jan. 30, 2021	Feb. 01, 2020	Feb. 02, 2019
Operating groups
Net sales	$ 748,833	$ 1,122,790	$ 1,107,466
Operating Segments | Tommy Bahama
Operating groups
Net sales	419,817	676,652	675,358
Operating Segments | Lilly Pulitzer
Operating groups
Net sales	231,078	284,700	272,299
Operating Segments | Southern Tide
Operating groups
Net sales	34,664	46,409	45,248
Operating Segments | Lanier Apparel
Operating groups
Net sales	38,796	95,200	99,904
Corporate and Other
Operating groups
Net sales	24,478	19,829	14,657
Retail
Operating groups
Net sales	$ 202,071	$ 440,803	$ 439,556
Net sales (as a percent)	27.00%	39.00%	40.00%
Retail | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	37.00%	48.00%	48.00%
Retail | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	20.00%	41.00%	42.00%
Retail | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	4.00%
E-commerce
Operating groups
Net sales	$ 323,900	$ 262,283	$ 239,034
Net sales (as a percent)	43.00%	23.00%	21.00%
E-commerce | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	36.00%	20.00%	18.00%
E-commerce | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	64.00%	38.00%	36.00%
E-commerce | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	32.00%	21.00%	18.00%
E-commerce | Corporate and Other
Operating groups
Net sales (as a percent)	63.00%	59.00%	54.00%
Restaurant
Operating groups
Net sales	$ 48,428	$ 83,836	$ 84,530
Net sales (as a percent)	7.00%	8.00%	8.00%
Restaurant | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	11.00%	12.00%	13.00%
Wholesale
Operating groups
Net sales	$ 173,209	$ 333,986	$ 341,615
Net sales (as a percent)	23.00%	30.00%	31.00%
Wholesale | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	16.00%	20.00%	21.00%
Wholesale | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	16.00%	21.00%	22.00%
Wholesale | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	64.00%	79.00%	82.00%
Wholesale | Operating Segments | Lanier Apparel
Operating groups
Net sales (as a percent)	100.00%	100.00%	100.00%
Wholesale | Corporate and Other
Operating groups
Net sales (as a percent)	33.00%	34.00%	30.00%
Other
Operating groups
Net sales	$ 1,225	$ 1,882	$ 2,731
Other | Corporate and Other
Operating groups
Net sales (as a percent)	4.00%	7.00%	16.00%