Exhibit 5

May 3, 2006

Imperial Sugar Company

One Imperial Square

8016 Highway 90-A

Sugar Land, Texas 77487

Ladies and Gentlemen:

As set forth in the Registration Statement on Form S-3 (the “Registration Statement”) filed on the date hereof by Imperial Sugar Company, a Texas corporation (the “Company”), with the Securities and Exchange Commission (the “Commission”) under the Securities Act of 1933, as amended (the “Act”), relating to the possible sale from time to time by the selling shareholder identified in the Registration Statement of up to 3,315,715 shares (the “Shares”) of common stock, without par value, of the Company, we are passing upon certain legal matters for the Company in connection with the Shares. At your request, we are furnishing this opinion to you for filing as Exhibit 5 to the Registration Statement.

In our capacity as your counsel in the connection referred to above, we have examined (i) the originals, or copies certified or otherwise identified, of the Amended and Restated Articles of Incorporation and the Amended and Restated By-Laws of the Company, each as amended to date, and (ii) the originals, or copies certified or otherwise identified, of corporate records of the Company, including minute books of the Company as furnished to us by the Company, certificates of public officials and of representatives of the Company, statutes and other instruments and documents as a basis for the opinion hereinafter expressed. In giving such opinion, we have relied upon certificates of officers of the Company and of public officials with respect to the accuracy of the material factual matters contained in such certificates.

We have assumed that all signatures on all documents examined by us are genuine, that all documents submitted to us as originals are accurate and complete, that all documents submitted to us as copies are true and correct copies of the originals thereof and that all information submitted to us was accurate and complete.

On the basis of the foregoing, and subject to the assumptions, limitations and qualifications set forth herein, we are of the opinion that the Shares have been duly authorized by all necessary corporate action on the part of the Company, and are validly issued, fully paid and nonassessable.

The opinion set forth above is based on and limited in all respects to matters of the federal laws of the United States and the laws of the State of Texas, each as currently in effect.

Imperial Sugar Company 2 May 3, 2006

We hereby consent to the filing of this opinion with the Commission as Exhibit 5 to the Registration Statement and to the reference to our Firm under the heading “Legal Matters” in the prospectus forming a part of the Registration Statement. In giving such consent, we do not admit that we are within the category of persons whose consent is required under Section 7 of the Act or the rules and regulations of the Commission thereunder.

Very truly yours,

/s/ Baker Botts L.L.P.