Operating Group Information - Sales by operating group (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Nov. 02, 2019	Nov. 03, 2018	Nov. 02, 2019	Nov. 03, 2018
Operating groups
Net sales	$ 241,221	$ 233,662	$ 825,194	$ 808,931
Operating Segments | Tommy Bahama
Operating groups
Net sales	127,023	123,130	480,623	482,990
Operating Segments | Lilly Pulitzer
Operating groups
Net sales	71,659	68,213	219,809	208,463
Operating Segments | Lanier Apparel
Operating groups
Net sales	29,377	29,037	76,871	72,806
Operating Segments | Southern Tide
Operating groups
Net sales	9,102	9,496	35,704	34,745
Corporate and Other
Operating groups
Net sales	4,060	3,786	12,187	9,927
Retail
Operating groups
Net sales	$ 83,636	$ 80,624	$ 324,892	$ 322,940
Net Sales (as a percent)	35.00%	35.00%	39.00%	40.00%
Retail | Operating Segments | Tommy Bahama
Operating groups
Net Sales (as a percent)	46.00%	46.00%	48.00%	48.00%
Retail | Operating Segments | Lilly Pulitzer
Operating groups
Net Sales (as a percent)	35.00%	35.00%	43.00%	44.00%
E-commerce
Operating groups
Net sales	$ 62,310	$ 56,392	$ 180,736	$ 164,277
Net Sales (as a percent)	26.00%	24.00%	22.00%	20.00%
E-commerce | Operating Segments | Tommy Bahama
Operating groups
Net Sales (as a percent)	14.00%	14.00%	18.00%	17.00%
E-commerce | Operating Segments | Lilly Pulitzer
Operating groups
Net Sales (as a percent)	55.00%	53.00%	36.00%	35.00%
E-commerce | Operating Segments | Lanier Apparel
Operating groups
Net Sales (as a percent)	1.00%		1.00%
E-commerce | Operating Segments | Southern Tide
Operating groups
Net Sales (as a percent)	19.00%	16.00%	18.00%	16.00%
E-commerce | Corporate and Other
Operating groups
Net Sales (as a percent)	57.00%	50.00%	60.00%	54.00%
Restaurant
Operating groups
Net sales	$ 17,325	$ 16,329	$ 61,457	$ 63,089
Net Sales (as a percent)	7.00%	7.00%	8.00%	8.00%
Restaurant | Operating Segments | Tommy Bahama
Operating groups
Net Sales (as a percent)	14.00%	13.00%	13.00%	13.00%
Wholesale
Operating groups
Net sales	$ 77,595	$ 79,654	$ 256,794	$ 256,432
Net Sales (as a percent)	32.00%	34.00%	31.00%	32.00%
Wholesale | Operating Segments | Tommy Bahama
Operating groups
Net Sales (as a percent)	26.00%	27.00%	21.00%	22.00%
Wholesale | Operating Segments | Lilly Pulitzer
Operating groups
Net Sales (as a percent)	10.00%	12.00%	21.00%	21.00%
Wholesale | Operating Segments | Lanier Apparel
Operating groups
Net Sales (as a percent)	99.00%	100.00%	99.00%	100.00%
Wholesale | Operating Segments | Southern Tide
Operating groups
Net Sales (as a percent)	81.00%	84.00%	82.00%	84.00%
Wholesale | Corporate and Other
Operating groups
Net Sales (as a percent)	36.00%	34.00%	32.00%	27.00%
Other
Operating groups
Net sales	$ 355	$ 663	$ 1,315	$ 2,193
Other | Corporate and Other
Operating groups
Net Sales (as a percent)	7.00%	16.00%	8.00%	19.00%