Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in registration statements on Form S-8 (File No. 333-177442 and No. 333-181924) of our report dated June 28, 2017, with respect to the financial statements and supplemental schedule of the Asheville Savings Bank Retirement Savings Plan included in this Annual Report on Form 11-K for the fiscal year ended December 31, 2016.

/s/ Dixon Hughes Goodman LLP

Atlanta, Georgia

June 28, 2017