QuickLinks -- Click here to rapidly navigate through this document

EXHIBIT 23

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in

(i)

Post-Effective Amendment No. 2 to the Registration Statement (Form S-8, No. 2-73803), which relates to the Phillips-Van Heusen Corporation Employee Savings and Retirement Plan,



(ii)

Registration Statement (Form S-8, No. 33-50841) and Registration Statement (Form S-8, No. 33-59602), each of which relate to the Phillips-Van Heusen Corporation Associates Investment Plan for Residents of the Commonwealth of Puerto Rico,



(iii)

Registration Statement (Form S-8, No. 33-59101), which relates to the Voluntary Investment Plan of Phillips-Van Heusen Corporation (Crystal Brands Division),



(iv)

Post-Effective Amendment No. 4 to Registration Statement (Form S-8, No. 2-72959), Post Effective Amendment No. 6 to Registration Statement (Form S-8, No. 2-64564), and Post Effective Amendment No. 13 to Registration Statement (Form S-8, No. 2-47910), each of which relate to the 1973 Employee's Stock Option Plan of Phillips-Van Heusen Corporation,



(v)

Registration Statement (Form S-8, No. 33-38698), Post-Effective Amendment No. 1 to Registration Statement (Form S-8, No. 33-24057) and Registration Statement (Form S-8, No. 33-60793), each of which relate to the Phillips-Van Heusen Corporation 1987 Stock Option Plan,



(vi)

Registration Statement (Form S-8, No. 333-29765) which relates to the Phillips-Van Heusen Corporation 1997 Stock Option Plan,



(vii)

Registration Statement (Form S-8, No. 333-41068) which relates to the Phillips-Van Heusen Corporation 2000 Stock Option Plan,



(viii)

Registration Statement (Form S-8, No. 333-109000), which relates to the Phillips-Van Heusen Corporation 2003 Stock Option Plan,



(ix)

Registration Statement (Form S-8, No. 333-125694) which relates to the Phillips-Van Heusen Corporation Associates Investment Plan for Salaried Associates, Associates Investment Plan for Hourly Associates and Associates Investment Plan for Residents of the Commonwealth of Puerto Rico, and

of Phillips-Van Heusen Corporation and in the related Prospectuses of our reports dated March 20, 2006 with respect to the consolidated financial statements and financial statement schedule of Phillips-Van Heusen Corporation, Phillips-Van Heusen Corporation's management's assessment of the effectiveness of internal control over financial reporting, and the effectiveness of internal control over financial reporting of Phillips-Van Heusen Corporation, included in this Annual Report (Form 10-K) for the year ended January 29, 2006.

ERNST & YOUNG LLP

New York, New York
April 11, 2006

QuickLinks

Consent of Independent Registered Public Accounting Firm