Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement (Form S-8 No. 33-46810) pertaining to the Phillips-Van Heusen Corporation Associates Investment Plan for Hourly Associates of our report dated June 10, 2004, with respect to the financial statements of the Phillips-Van Heusen Corporation Associates Investment Plan for Hourly Associates included in this Annual Report (Form 11-K) for the year ended December 31, 2003.

/s/Ernst & Young LLP

June 22, 2004