Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statements (Forms S-3 Nos. 333-118966, 333-112598, 333-90742, 333-160379, and 33-53358 and Forms S-8 Nos. 333-112561, 333-112218, 333-136890, 33-52892, 333-164618, 333-163245, 333-170708, and 333-170764) of Casual Male Retail Group, Inc. and in the related Prospectus of our reports dated March 16, 2012, with respect to the consolidated financial statements of Casual Male Retail Group, Inc. and the effectiveness of internal control over financial reporting of Casual Male Retail Group, Inc., included in this Annual Report (Form 10-K) for the year ended January 28, 2012.

/s/ Ernst & Young LLP

Boston, Massachusetts

March 16, 2012