Exhibit 32.1

Certification Pursuant to

18 U.S.C. Section 1350,

As Adopted Pursuant to Section 906

Of the Sarbanes-Oxley Act of 2002

In connection with the Annual Report on Form 11-K of the Casual Male Retail Group, Inc. 401(k) Salary Saving Plan (the “Plan”) for the period ending December 31, 2005, as filed with the Securities and Exchange Commission on the date hereof (the “Report”), I, Dennis R. Hernreich, Executive Vice President, Chief Financial Officer, Chief Operating Officer of Casual Male Retail Group, Inc., the Plan Administrator, certify pursuant to 18 U.S.C. Section 1350, as adopted pursuant to Section 906 of the Sarbanes-Oxley Act of 2002, that:

(1) The Report fully complies with the requirements of Section 15(d) of the Securities Exchange Act of 1934, as amended (the “Exchange Act”); and

(2) The information contained in the Report fairly presents, in all material respects, the net assets available for benefits, and changes in net assets available for benefits of the Plan.

A signed original of this written statement required by Section 906, or other document authenticating, acknowledging, or otherwise adopting the signature that appears in typed form within the electronic version of this written statement required by Section 906, has been provided to Casual Male Retail Group, Inc. and will be retained by Casual Male Retail Group, Inc. and furnished to the Securities and Exchange Commission or its staff upon request.

/s/ Dennis R. Hernreich

Executive Vice President, Chief Financial Officer, Chief Operating Officer of Casual Male Retail Group, Inc., the Plan Administrator

June 23, 2006