Exhibit 10.4

AMENDMENT TO CONSULTING AGREEMENT

Dated as of May 17, 2009

WHEREAS, Casual Male Retail Group, Inc. (formerly Designs, Inc., the “Corporation”) and Jewelcor Management, Inc. (the “Independent Contractor”) entered into a certain Consulting Agreement dated April 29, 2000, as amended by Letter Agreement dated April 28, 2001, by Letter Agreement dated April 28, 2002, by Amendment to Consulting Agreement dated April 29, 2003, by Amendment to Consulting Agreement dated April 26, 2004, by Amendment to Consulting Agreement dated August 26, 2004, by Amendment to Consulting Agreement dated June 15, 2005, as amended by Letter Agreement dated May 26, 2006, by Letter Agreement dated April 29, 2007 and by Amendment to Consulting Agreement dated September 8, 2008 (hereinafter referred to as “the Agreement”), and

WHEREAS, Corporation and Independent Contractor wish to amend, modify and/or restate certain terms, provisions, conditions and covenants of the Agreement.

NOW THEREFORE, for and in consideration of the foregoing, the mutual promises and covenants set forth in this Agreement, and for other good and valuable consideration, the receipt and sufficiency of which are hereby expressly acknowledged, the Corporation and Independent Contractor hereby agree to amend the Agreement as follows:

1. Term of Agreement.

The term of the Agreement shall expire on April 29, 2011. The term can be further extended only by agreement of both the Corporation and Independent Contractor. For the sake of clarity and the avoidance of doubt, the parties hereby agree and acknowledge that the term of the Agreement as set forth in SECTION FIVE, headed “Duration”, of the original agreement dated April 29, 2000, and as amended through September 8, 2008 as set forth above, is deemed replaced in its entirety by this Amendment to Consulting Agreement.

2. Compensation. The first paragraph under “Compensation” in the Amendment to Consulting Agreement dated April 29, 2007 shall be amended by adding the following sentence:

For fiscal year 2009, the compensation to be paid under the Agreement shall consist of a 10.0% decrease in each payment made on or after the effective date of this Agreement.

THE REMAINING terms of the Agreement as amended shall remain in full force and effect without change.

IN WITNESS WHEREOF, the parties hereto have executed this Amendment to Consulting Agreement as a sealed instrument, in any number of counterpart copies, each of which shall be deemed an original for all purposes, as of the day and year first written above.

JEWELCOR MANAGEMENT, INC.
By:	/s/    Seymour Holtzman
	Seymour Holtzman
	August 27, 2009
CASUAL MALE RETAIL GROUP, INC.
By:	/s/    David A. Levin
	David A. Levin
	August 4, 2009
By:	/s/    Dennis R. Hernreich
	Dennis R. Hernreich
	August 4, 2009