Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in Registration Statements Nos. 333-167264, 333-145043, 333-139017, and 333-52148 on Forms S-8 and Registration Statements Nos. 333-161165 and 333-161893 on Forms S-3 of our report dated March 15, 2012 on the consolidated financial statements of Hudson Valley Holding Corp. and Subsidiaries and the effectiveness of internal control over financial reporting of Hudson Valley Holding Corp. and Subsidiaries, which report is included in this Annual Report on Form 10-K for Hudson Valley Holding Corp. and Subsidiaries for the year ended December 31, 2011.

/s/ Crowe Horwath LLP

New York, New York

March 15, 2012