Exhibit 23.2

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement (Form S-8, Reg. No. 333-112218) of Casual Male Retail Group, Inc. of our report dated June 26, 2008, with respect to the statement of net assets available for benefits as of December 31, 2007 of the Casual Male Retail Group, Inc. 401(k) Hourly Savings Plan included in this Annual Report (Form 11-K) for the year ended December 31, 2008.

/s/ Mahoney Cohen & Company, CPA P.C.

New York, New York

June 29, 2009