Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement (Form S-8) pertaining to the Harley-Davidson Retirement Savings Plan for Salaried Employees, the Harley-Davidson Retirement Savings Plan for Milwaukee and Tomahawk Hourly Bargaining Unit Employees, the Harley-Davidson Retirement Savings Plan for Kansas City Hourly Bargaining Unit Employees, the Harley-Davidson Retirement Savings Plan for York Hourly Bargaining Unit Employees and the Harley-Davidson Financial Services, Inc. 401(k) Profit Sharing Plan of our reports (a) dated February 23, 2012, with respect to the consolidated financial statements and schedule of Harley-Davidson, Inc., and the effectiveness of internal control over financial reporting of Harley-Davidson, Inc., included in its Annual Report (Form 10-K) for the year ended December 31, 2011, and (b) dated June 24, 2011, with respect to the financial statements and schedule of the Harley-Davidson Retirement Savings Plan for Salaried Employees, the Harley-Davidson Retirement Savings Plan for Milwaukee and Tomahawk Hourly Bargaining Unit Employees, the Harley-Davidson Retirement Savings Plan for Kansas City Hourly Bargaining Unit Employees and the Harley-Davidson Retirement Savings Plan for York Hourly Bargaining Unit Employees included in the Plans’ Annual Report (Form 11-K), for the year ended December 31, 2010, filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

Milwaukee, Wisconsin

May 30, 2012