Exhibit (a)(1)(E)

Offer To Purchase For Cash

All Outstanding Shares of Common Stock

of

Vitesse Semiconductor Corporation

at

$5.28 NET PER SHARE

Pursuant to the Offer to Purchase dated March 31, 2015

by

LLIU100 Acquisition Corp.

a wholly owned subsidiary of

Microsemi Corporation

THE OFFER AND WITHDRAWAL RIGHTS WILL EXPIRE AT 12:00 MIDNIGHT, NEW YORK CITY

TIME, AT THE END OF APRIL 27, 2015,

UNLESS THE OFFER IS EXTENDED.

March 31, 2015

To Brokers, Dealers, Commercial Banks, Trust Companies and Other Nominees:

We have been engaged by LLIU100 Acquisition Corp., a Delaware corporation (the “Purchaser”) and a wholly owned subsidiary of Microsemi Corporation, a Delaware corporation, to act as Information Agent in connection with the Purchaser’s offer to purchase (the “Offer”) all outstanding shares of common stock, par value $0.01 per share (the “Company Shares”), of Vitesse Semiconductor Corporation, a Delaware corporation (the “Company”), at a purchase price of $5.28 per Company Share, net to the tendering stockholder in cash, without interest and less any required withholding taxes, upon the terms and subject to the conditions set forth in the Offer to Purchase, dated March 31, 2015 (together with any amendments and supplements thereto, the “Offer to Purchase”), and the related Letter of Transmittal enclosed herewith.

For your information and for forwarding to your clients for whom you hold Company Shares registered in your name or in the name of your nominee, we are enclosing the following documents:

1. The Offer to Purchase;

2. The Letter of Transmittal for your use in accepting the Offer and tendering Company Shares and for the information of your clients, together with an IRS Form W-9 with instructions providing information relating to backup federal income tax withholding;

3. A notice of guaranteed delivery to be used to accept the Offer if the certificate(s) for the Company Shares (“Share Certificates”) and all other required documents cannot be delivered to Computershare (the “Depositary”) by the Expiration Date (as defined in the Offer to Purchase) or if the procedure for book-entry transfer cannot be completed by the Expiration Date (the “Notice of Guaranteed Delivery”);

4. A form of letter which may be sent to your clients for whose accounts you hold Company Shares registered in your name or in the name of your nominee, with space provided for obtaining such clients’ instructions with regard to the Offer;

5. The Company’s Solicitation/Recommendation Statement on Schedule 14D-9, dated March 31, 2015; and

6. A return envelope addressed to the Depositary for your use only.

1

Certain conditions to the Offer are described in Section 15 of the Offer to Purchase.

We urge you to contact your clients as promptly as possible. Please note that the Offer will expire at the end of April 27, 2015, unless the Offer is extended. Previously tendered Company Shares may be withdrawn at any time until the Offer has expired and, if the Purchaser has not accepted such Company Shares for payment by the end of May 29, 2015, such Company Shares may be withdrawn at any time after that date until the Purchaser accepts Company Shares for payment.

For Company Shares to be properly tendered pursuant to the Offer, (a) the Share Certificates or confirmation of receipt of such Company Shares under the procedure for book-entry transfer, together with a properly completed and duly executed Letter of Transmittal, including any required signature guarantees, or an “Agent’s Message” (as defined in the Offer to Purchase) in the case of book-entry transfer, and any other documents required in the Letter of Transmittal, must be timely received by the Depositary or (b) the tendering stockholder must comply with the guaranteed delivery procedures, all in accordance with the Offer to Purchase and Letter of Transmittal. You may gain some additional time by making use of the Notice of Guaranteed Delivery. Company Shares tendered by the Notice of Guaranteed Delivery will be excluded from the calculation of the Minimum Condition (as defined in the Offer to Purchaser), unless such Company Shares and other required documents are received by the Depositary by the Expiration Date.

The Purchaser will not pay any fees or commissions to any broker or dealer or other person (other than the Depositary and the Information Agent as described in the Offer to Purchase) for soliciting tenders of Company Shares pursuant to the Offer. The Purchaser will, however, upon request, reimburse brokers, dealers, commercial banks and trust companies for reasonable and necessary costs and expenses incurred by them in forwarding materials to their customers. The Purchaser will pay all stock transfer taxes applicable to its purchase of Company Shares pursuant to the Offer, subject to Instruction 6 of the Letter of Transmittal.

Any inquiries you may have with respect to the Offer should be addressed to, and additional copies of the enclosed materials may be obtained from, the Information Agent (as defined in the Offer to Purchase) or the undersigned at the addresses and telephone numbers set forth on the back cover of the Offer to Purchase.

Very truly yours,

Okapi Partners

Nothing contained herein or in the enclosed documents shall render you the agent of the Purchaser, the Information Agent or the Depositary or any affiliate of any of them or authorize you or any other person to use any document or make any statement on behalf of any of them in connection with the Offer other than the enclosed documents and the statements contained therein.

The Information Agent for the Offer is:

437 Madison Avenue, 28^th Floor

New York, New York 10022

Banks and Brokerage Firms, Please Call: (212) 297-0720

Stockholders and All Others, Call Toll Free: (877) 566-1922

Email: info@okapipartners.com

2