Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statements (Form S-3 Nos. 333-118966, 333-112598, 333-90742, and 33-53358 and Form S-8 Nos. 333-112561, 333-112218, and 33-52892) of our reports dated April 7, 2005, with respect to the consolidated financial statements and schedule of Casual Male Retail Group, Inc., Casual Male Retail Group, Inc.’s management’s assessment of the effectiveness of internal control over financial reporting, and the effectiveness of internal control over financial reporting of Casual Male Retail Group, Inc., included in this Annual Report (Form 10-K) for the year ended January 29, 2005.

/s/ Ernst & Young LLP

Boston, Massachusetts

April 7, 2005