Exhibit 23.2

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement on Form S-8 pertaining to The Dixie Group, Inc. Amended and Restated 2006 Stock Awards Plan of our reports dated March 3, 2010, with respect to the consolidated financial statements and schedule of The Dixie Group, Inc. and the effectiveness of internal control over financial reporting of The Dixie Group, Inc. included in its Annual Report (Form 10-K) for the year ended December 26, 2009, filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

Atlanta, Georgia

July 30, 2010