Exhibit 23.1


            Consent of Independent Registered Public Accounting Firm

We consent to the reference to our firm under the caption "Experts" in the
Registration Statement (Form S-3) and related Prospectus of Casual Male Retail
Group, Inc. for the registration of 1,818,504 shares of Casual Male Retail
Group, Inc. common stock and to the incorporation by reference therein of our
report dated March 24, 2004, with respect to the consolidated financial
statements and schedules of Casual Male Retail Group, Inc. included in its
Annual Report (Form 10-K) for the year ended January 31, 2004, filed with the
Securities and Exchange Commission.

                                                      ERNST & YOUNG LLP

September 9, 2004