Exhibit 5.1


                                     November 22, 2004

Casual Male Retail Group, Inc.
555 Turnpike Street
Canton, Massachusetts 02021

            Re:   Registration Statement on Form S-3
                  ----------------------------------

Ladies and Gentlemen:

      We have acted as special counsel to Casual Male Retail Group, Inc., a
Delaware corporation (the "Company"), in connection with a Registration
Statement on Form S-3, as amended (the "Registration Statement"), for the
registration of the sale from time to time by the holders thereof of 1,818,504
shares (the "Shares") of common stock, par value $0.01 per share, of the
Company.

      We have made such inquiries and reviewed such documents and records as we
have deemed necessary to enable us to express an opinion on the matters covered
hereby, and we have also examined and relied upon representations, statements or
certificates of public officials and officers and representatives of the
Company.

      Based upon and subject to the foregoing, we are of the opinion that:

1.    the issued Shares have been duly authorized and legally issued and are
      fully paid and non-assessable; and

2.    the Shares that have not yet been issued have been duly authorized and,
      when issued in accordance with the terms applicable thereto, will be
      legally issued, fully paid and non-assessable.

      We do not express any opinion with respect to any law other than the laws
of the State of New York, the Delaware General Corporation Law and the federal
laws of the United States of America. This opinion is rendered only with respect
to the laws and legal interpretations and the facts and circumstances in effect
on the date hereof.

      We hereby consent to the use of this opinion as Exhibit 5.1 to the
Registration Statement and to the use of our name under the caption "Legal
Matters" in the prospectus included in the Registration Statement. In giving
this consent, we do not thereby admit that we are within the category of persons
whose consent is required under Section 7 of the Securities Act of 1933, as
amended, or the rules and regulations promulgated thereunder.


                                    Very truly yours,
                                    /s/ Kramer Levin Naftalis & Frankel LLP
                                    Kramer, Levin, Naftalis & Frankel LLP