Exhibit 23.1

Consent of Independent Auditors

We consent to the incorporation by reference in the Registration Statements (Form S-3 No. 333-90742 and Form S-8 Nos. 33-22957, 33-32690, 33-32687 and 33-52892) of Casual Male Retail Group, Inc. (formerly Designs, Inc.) of our report dated March 26, 2003, with respect to the consolidated financial statements and schedule of Casual Male Retail Group, Inc. included in this Annual Report (Form 10-K) for the year ended February 1, 2003.

ERNST & YOUNG LLP

Boston, Massachusetts

May 5, 2003