Exhibit 23.1


Consent of Independent Auditors

We consent to the reference to our firm under the caption "Experts" in the
Registration Statement (Form S-3) and related Prospectus of Casual Male Retail
Group, Inc. for the registration of $100,000,000 principal amount of 5%
convertible senior subordinated notes due 2024 and shares of Casual Male Retail
Group, Inc. common stock issuable upon conversion of the notes and to the
incorporation by reference therein of our report dated March 26, 2003, with
respect to the consolidated financial statements and schedules of Casual Male
Retail Group, Inc. included in its Annual Report (Form 10-K) for the year ended
February 1, 2003, filed with the Securities and Exchange Commission.

                                                      ERNST & YOUNG LLP

February 5, 2004