EXHIBIT 23.2

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the following Registration Statements:

(1) Registration Statement (Form S-3 No. 333-133783) of Imperial Sugar Company,

(2) Registration Statement (Form S-8 No. 333-131930) pertaining to the Imperial Sugar Company Long Term Incentive Plan of Imperial Sugar Company,

(3) Registration Statement (Form S-8 No. 333-105160) pertaining to the Imperial Sugar Company Long Term Incentive Plan of Imperial Sugar Company, and

(4) Registration Statement (Form S-8 No. 333-102876) pertaining to the Imperial Sugar Company Long Term Incentive Plan of Imperial Sugar Company;

of our report dated January 5, 2012, with respect to the financial statements of Louisiana Sugar Refining, LLC for the year ended September 30, 2011 (not presented separately herein), included in this Annual Report (Form 10-K) of Imperial Sugar Company for the year ended September 30, 2011.

/s/ Ernst & Young LLP

New Orleans, Louisiana

January 5, 2012