Exhibit 5.1

September 2, 2009

Casual Male Retail Group, Inc.

555 Turnpike Street

Canton, Massachusetts 02021

Re: Casual Male Retail Group, Inc./Registration Statement on Form S-3

Ladies and Gentlemen:

As legal counsel to Casual Male Retail Group, Inc., a Delaware corporation (the “Company”), we have assisted in the preparation of the Company’s Registration Statement on Form S-3 (File No. 333-160379) (the “Registration Statement”), filed with the Securities and Exchange Commission (the “Commission”) on July 1, 2009 pursuant to the Securities Act of 1933, as amended (the “Securities Act”), declared effective by the Commission on August 3, 2009, and the prospectus supplement dated September 1, 2009, filed with the Commission under Rule 424(b) of the Securities Act (the “Prospectus Supplement”), relating to the proposed public offering (the “Offering”) of up to 4,950,000 shares of the Company’s Common Stock (the “Shares”).   With respect to the opinions set forth below, we have examined originals, certified copies, or copies otherwise identified to our satisfaction as being true copies, only of the following:   A.     The Certificate of Incorporation of the Company, as amended to date;   B.     The Bylaws of the Company, as amended to date;   C.     The Registration Statement;   D.     The Prospectus Supplement;   E.     The subscription agreements between the investors in the Offering and the Company (the “Subscription Agreements”);   F.      The resolutions of the Board of Directors of the Company relating to the approval of the filing of the Registration Statement and transactions in connection therewith, including the Offering; and   G.     Such other documents, records and other instruments as we have deemed appropriate for purposes of the opinion set forth herein.

In rendering the opinions set forth below, we have assumed the genuineness of all signatures, the legal capacity of natural persons, the authenticity of all documents submitted to us as originals, the conformity to original documents of all documents submitted to us as duplicates or certified or conformed copies, and the authenticity of originals or such latter documents. We have also assumed for purposes of our opinion that the Subscription Agreements have been duly authorized, executed and delivered by the parties thereto other than the Company and constitute a legal, valid and binding obligation of each such party, and that such parties have the requisite organizational and legal power to perform their obligations under the Subscription Agreements.

Based solely upon and subject to the foregoing, and subject to the assumptions, limitations, and qualifications stated herein, we are of the opinion that the Shares have been duly authorized and, when issued and sold in the manner and under the terms described in the Registration Statement, Prospectus Supplement and the Subscription Agreements, will be validly issued, fully paid and nonassessable.

We express no opinion as to the applicability or effect of any laws, orders, or judgments of any state or other jurisdiction other than federal securities laws and the substantive laws of the state of Delaware. Further, our opinion is based solely upon existing laws, rules, and regulations, and we undertake no obligation to advise you of any changes that may be brought to our attention after the date hereof.

We understand that this opinion is to be used in connection with the Registration Statement. We hereby consent to the filing of this opinion letter as Exhibit 5.1 to the Company’s Current Report on Form 8-K filed as of the date hereof and to the reference to this firm under the caption “Legal Matters” in the Prospectus Supplement constituting a part of the Registration Statement. In giving this consent, we do not admit that we are experts within the meaning of Section 11 of the Act or within the category of persons whose consent is required by Section 7 of the Act.

Very truly yours,

/s/ Greenberg Traurig, LLP

Greenberg Traurig, LLP

- 2 -