Exhibit 16

November 15, 2013

Securities and Exchange Commission

100 F Street, N.E.

Washington, DC 20549

Ladies and Gentlemen:

We have read Item 4.01 of Form 8-K dated November 15, 2013, of The Dixie Group, Inc. and are in agreement with the statements contained in the third, fourth and fifth paragraphs on page 2 therein. We have no basis to agree or disagree with other statements of the registrant contained therein.

Very truly yours,

/s/ Ernst & Young LLP