Exhibit 10.3

July 11, 2007

Casual Male Retail Group, Inc.

555 Turnpike Street

Canton, Massachusetts 02021

Attention: Dennis Heinreich

Chief Financial Officer

Re: Fifth Amended and Restated Loan and Security Agreement.

Dear Mr. Heinreich:

Reference is made to a certain Fifth Amended and Restated Loan and Security Agreement dated December 20, 2006 by and among others Bank of America, N.A., as Agent, and Casual Male Retail Group, Inc., as Borrower (the “Loan Agreement”). Please be advised that the definition of Availability set forth in the Loan Agreement is amended by deleting clause (a)(iv) therefrom so as not to deduct Availability Reserves from the Revolving Credit Commitments when determining Availability. Notwithstanding the foregoing, Availability Reserves continue to be deducted from the Borrowing Base when calculating Availability.

Please acknowledge your acceptance of the foregoing Amendment by endorsing a copy of this letter and returning it to the undersigned.

Very truly yours,

BANK OF AMERICA, N.A.

/s/    Peter A. Foley        

Peter A. Foley

ACKNOWLEDGED AND ACCEPTED

CASUAL MALE RETAIL GROUP, INC.

/s/ Dennis R. Hernreich

Dennis Hernreich

Chief Financial Officer