EXHIBIT 5.1

[DAVIS POLK & WARDWELL LETTERHEAD]

December 29, 2004

Re: Registration Statement on Form S-3

Vitesse Semiconductor Corporation

741 Calle Plano

Camarillo, California 93012

Ladies and Gentlemen:

We have acted as counsel for Vitesse Semiconductor Corporation, a Delaware corporation (the “Company”) in connection with the preparation of the registration statement on Form S-3 (the “Registration Statement”) filed by the Company with the Securities and Exchange Commission pursuant to the Securities Act of 1933, as amended (the “Securities Act”), registering the sale from time to time of (i) $96,700,000 aggregate principal amount of 1.50% Convertible Subordinated Debentures due 2024 (the “Debentures”) and (ii) such indeterminate number of shares of common stock, par value $0.01 per share, of the Company as may be issuable upon conversion of the Debentures (the “Underlying Securities”). The Debentures have been issued pursuant to the Indenture dated as of September 22, 2004, (the “Indenture”) between the Company and U.S. Bank National Association, as trustee. The Debentures and the Underlying Securities are to be offered and sold from time to time by their holders.

We, as your counsel, have examined originals or copies, certified or otherwise identified to our satisfaction, of such documents, corporate records, certificates of public officials and other instruments as we have deemed necessary for the purposes of rendering this opinion.

On the basis of the foregoing, we advise you that, in our opinion:

1. The Debentures have been duly authorized, executed and delivered by the Company and are legal, valid and binding obligations of the Company, enforceable in accordance with their terms, subject to applicable bankruptcy, insolvency and similar laws affecting the enforcement of creditors’ rights generally and equitable principles of general applicability.

2. The Underlying Securities reserved for issuance upon conversion of the Debentures have been duly authorized and reserved and, when issued upon conversion of the Debentures in accordance with the terms of the Debentures and the Indenture, will be validly issued, fully paid and non-assessable.

Our opinion in paragraph (1) above is subject to the qualification that we express no opinion as to the validity, legally binding effect or enforceability of any provisions of the Debentures that require or relate to premiums or liquidated damages at a rate or in an amount that a court would determine under applicable law to be commercially unreasonable or to constitute a penalty or forfeiture.

We are members of the Bars of the States of New York and California and the foregoing opinion is limited to the laws of the States of New York and California and the General Corporation Law of the State of Delaware.

We hereby consent to the filing of this opinion as an exhibit to the Registration Statement and further consent to the reference to our name under the caption “Legal Matters” in the prospectus, which is a part of the Registration Statement. In giving this consent, we do not admit that we are in the category of persons whose consent is required under Section 7 of the Securities Act.

Sincerely,

/s/ Davis Polk & Wardwell

2