Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement (Form S-8, Reg. No. 333-112218) pertaining to the Casual Male Retail Group, Inc. 401 (k) Salaried Savings Plan of Casual Male Retail Group, Inc of our report dated June 12, 2006, with respect to the financial statements and schedules of the Casual Male Retail Group, Inc. 401(k) Salaried Savings Plan included in this Annual Report on (Form 11-K) for the year ended December 31, 2005.

/s/ Mahoney Cohen & Company, CPA, P.C

New York, New York

June 23, 2006