Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statement (No. 333-112218) on Form S-8 of our report dated June 28, 2012, relating to the statements of net assets available for benefits of Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan as of December 31, 2011, and 2010, the related statements of changes in net assets available for benefits for the year ended December 31, 2011, and the related supplemental schedule as of December 31, 2011, which report appears in the December 31, 2011 annual report on Form 11-K of Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan.

/s/ Mayer Hoffman McCann CPAs

(The New York Practice of Mayer Hoffman McCann P.C.)

New York, New York

June 28, 2012