Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statement (No. 333-112218) on Form S-8 of our report dated June 28, 2013, relating to the statements of net assets available for benefits of Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan as of December 31, 2012, and 2011, the related statement of changes in net assets available for benefits for the year ended December 31, 2012, and the related supplemental schedules as of and for the year ended December 31, 2012, which reports appear in the December 31, 2012 annual report on Form 11-K of Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan.

/s/ Mayer Hoffman McCann CPAs

(The New York Practice of Mayer Hoffman McCann P.C.)

New York, New York

June 28, 2013