Exhibit 23.1



                         CONSENT OF INDEPENDENT AUDITORS



We consent to the reference to our firm under the caption "Experts" in the
Registration Statement (Form S-3) and related Prospectus of Casual Male Retail
Group, Inc. for the registration of $100,000,000 principal amount of 5%
convertible senior subordinated notes due 2024 and shares of Casual Male Retail
Group, Inc. common stock issuable upon conversion of the notes and to the
incorporation by reference therein of our report dated March 24, 2004, with
respect to the consolidated financial statements and schedule of Casual Male
Retail Group, Inc. included in its Annual Report (Form 10-K) for the year ended
January 31, 2004, filed with the Securities and Exchange Commission.



                                                     /s/ ERNST & YOUNG LLP


Boston, Massachusetts
April 20, 2004