EXHIBIT 23

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in Registration Statement No. 333-105160 and No. 333-102876 of Imperial Sugar Company (“Imperial”) each on Form S-8 of our report dated December 3, 2004, appearing in this Annual Report on Form 10-K of Imperial for the year ended September 30, 2004.

DELOITTE & TOUCHE LLP

Houston, Texas

December 8, 2004