Exhibit 23.2

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the reference to our firm under the caption “Experts” in the Registration Statement (Form S-3) and related Prospectus of Casual Male Retail Group, Inc. for the registration of an indeterminate number of shares of common stock and preferred stock; an indeterminate number of depositary shares; an indeterminate number of warrants to purchase common stock, preferred stock, depositary shares and/or units; an indeterminate number of purchase contracts to purchase common stock, preferred stock, depositary shares or warrants; an indeterminate number of rights to purchase common stock, preferred stock, depositary shares or warrants; and an indeterminate number of units and to the incorporation by reference therein of our reports dated March 18, 2009, with respect to the consolidated financial statements of Casual Male Retail Group, Inc. and the effectiveness of internal control over financial reporting of Casual Male Retail Group, Inc. included in its Annual Report (Form 10-K), as amended, for the fiscal year ended January 31, 2009, filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

Boston, Massachusetts

July 1, 2009