Exhibit 23

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statements (Form S-8 No.’s 333-134779, 333-89994, 33-59564, 333-87534, 333-81163, 333-80971, 333-118504) of The Dixie Group, Inc. of our reports dated March 4, 2008 with respect to the consolidated financial statements and schedule of The Dixie Group, Inc., and the effectiveness of internal control over financial reporting of The Dixie Group, Inc. included in the Annual Report (Form 10-K) for the year ended December 29, 2007.

/s/ Ernst & Young LLP

Atlanta, Georgia

March 4, 2008