Exhibit 23

Report of Independent Registered Public Accounting Firm

The Board of Directors and Shareholders

O’Charley’s Inc.:

We consent to the incorporation by reference in the registration statements No. 333-159265, No. 333-151107, No. 33-69934, No. 333-59484, No. 333-126221, and No. 333-129783 on Form S-8 of O’Charley’s Inc. of our reports dated February 29, 2012, with respect to the consolidated balance sheets of O’Charley’s Inc. and subsidiaries as of December 25, 2011 and December 26, 2010, and the related consolidated statements of operations, shareholders’ equity and comprehensive loss and cash flows for each of the fiscal years in the three-year period ended December 25, 2011 and the related financial statement schedule, and the effectiveness of internal control over financial reporting as of December 25, 2011, which reports appear in the 2011 annual report on Form 10-K of O’Charley’s Inc.

/s/ KPMG LLP

Nashville, Tennessee

February 29, 2012