Exhibit 18.1

PREFERABILITY LETTER FROM INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

August 18, 2006

Mr. Dennis Hernreich

Executive Vice President, Chief Operating Officer and Chief Financial Officer

Casual Male Retail Group, Inc.

55 Turnpike Street

Canton, Massachusetts 02021

Dear Mr. Hernreich:

Note 2 of the Notes to the Consolidated Financial Statements of Casual Male Retail Group, Inc. included in its Form 10-Q for the quarter ended July 29, 2006 describes a change in the method of accounting for merchandise inventory from the FIFO retail inventory method to the weighted-average cost method. There are no authoritative criteria for determining a “preferable” method of determining the cost of merchandise inventory based on the particular circumstances; however, we conclude that such change in the method of accounting is to an acceptable alternative method which, based on your business judgment to make this change and for the stated reasons, is preferable in your circumstances. We have not conducted an audit in accordance with the standards of the Public Company Accounting Oversight Board (United States) of any financial statements of the Company as of any date or for any period subsequent to January 28, 2006, and therefore we do not express any opinion on any financial statements of Casual Male Retail Group, Inc. subsequent to that date.

Very truly yours,

/s/ Ernst & Young LLP

Boston, Massachusetts

August 18, 2006