Exhibit 10.23

                              As of July 1, 2001

Mr. Alan I. Weinstein
Chairman and Chief Executive Officer
Casual Male Corp.
555 Turnpike Street
Canton, MA  02021

Dear Alan:

     Reference is made to the Forgivable Promissory Note dated July 8, 1998, as
amended as of December 12, 2000 (the "Note") made in connection with the
issuance of the Restricted Stock Award granted to you on July 8, 1998 .  Terms
not otherwise defined herein shall be used as defined in the Note.

     By this letter, the Board of Directors of Casual Male Corp. (the "Company")
and you hereby agree to modify the Note as described herein:

1.  Forgiveness Schedule.  The second paragraph of the Note is deleted in its
    --------------------
entirety and replaced with the following:

     "If Payor remains employed by Lender, or by any company directly or
     indirectly controlled by Lender through the following dates, the
     outstanding Principal Amount of this Note (the "Note") shall be
     automatically reduced to the amount indicated:


        --------------------------------------------------------------
         Date                              Outstanding Principal
         ----                              ---------------------
                                           Amount
                                           ------
        --------------------------------------------------------------
         Prior to 1/st/ year               $509,000
         anniversary from July
         8, 1998
        --------------------------------------------------------------
         1/st/ year anniversary            $407,200
         from July 8, 1998
        --------------------------------------------------------------
         2nd year anniversary              $305,400
         from July 8, 1998
        --------------------------------------------------------------
         5th year anniversary              $0
         from July 8, 1998
        --------------------------------------------------------------


     2. No Other Modifications. All terms and provisions of the Award and the
        ----------------------
        Note not otherwise modified herein shall continue in full force and
        affect.


     Kindly acknowledge your acceptance of these modifications and your
agreement thereto by executing two copies of this letter in the designated space
below. Please return one copy to Michael O'Hara and retain one for your records.


                                       Very truly yours,


                                       /s/  J. Christopher Clifford
                                       ----------------------------
                                       J. Christopher Clifford
                                       Chairman, Compensation Committee
                                       Casual Male Corp. Board of Directors



ACCEPTED AND AGREED TO:



by:  /s/  Alan I. Weinstein
  -------------------------
          Alan I. Weinstein



JCC:moh


Cc:  Michael A. O'Hara, Esq.
     Secretary of the Board