Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statement (No. 333-112218) on Form S-8 of Casual Male Retail Group, Inc. of our report dated June 29, 2011, relating to the financial statements and supplemental schedule of the Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan included in the December 31, 2010 Annual Report on Form 11-K for the Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan.

/s/ Mayer Hoffman McCann CPAs

(The New York Practice of Mayer Hoffman McCann P.C.)

New York, New York

June 29, 2011