Exhibit 5.1


                                               March 4, 2004


Casual Male Retail Group, Inc.
555 Turnpike Street
Canton, Massachusetts 02021


            Re:   Registration Statement on Form S-3
                  ----------------------------------

Ladies and Gentlemen:

      We have acted as special counsel to Casual Male Retail Group, Inc., a
Delaware corporation (the "Company"), in connection with a Registration
Statement on Form S-3 (the "Registration Statement"), for the registration of
the sale from time to time by the holders thereof of (i) $100,000,000 aggregate
principal amount of the Company's 5% Convertible Subordinated Notes Due 2024
(the "Notes") issued pursuant to the terms of an Indenture, the form of which is
filed as Exhibit 4.1 to the Registration Statement (the "Indenture"), and (ii)
the shares (the "Shares") of common stock, par value $0.01 per share, of the
Company initially issuable upon conversion of the Notes.

      We have made such inquiries and reviewed such documents and records as we
have deemed necessary to enable us to express an opinion on the matters covered
hereby, and we have also examined and relied upon representations, statements or
certificates of public officials and officers and representatives of the
Company. We have assumed that the Notes have been duly authenticated by the
Trustee for the Notes as provided for in the Indenture.

      Based upon and subject to the foregoing, we are of the opinion that:

      1. the Notes have been legally issued and constitute the binding
obligations of the Company, subject to applicable bankruptcy, insolvency and
other similar laws relating to or affecting the rights and remedies of creditors
generally and to general principles of equity; and

      2. the Shares have been duly authorized and reserved and, when issued upon
conversion of the Notes in accordance with the terms of the Notes and the
Indenture, will be legally issued, fully paid and non-assessable.

      We do not express any opinion with respect to any law other than the laws
of the State of New York, the Delaware General Corporation Law and the federal
laws of the United States of America. This opinion is rendered only with respect
to the laws and legal interpretations and the facts and circumstances in effect
on the date hereof.

      We hereby consent to the use of this opinion as Exhibit 5.1 to the
Registration Statement and to the use of our name under the caption "Legal
Matters" in the prospectus included in the



Registration Statement. In giving this consent, we do not thereby admit that we
are within the category of persons whose consent is required under Section 7 of
the Securities Act of 1933, as amended, or the rules and regulations promulgated
thereunder.


                                    Very truly yours,

                                    Kramer, Levin, Naftalis & Frankel LLP