Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in Registration Statement Nos. 333-179164 and 333-143622 on Form S-8 of our reports dated March 7, 2012, relating to the consolidated financial statements and financial statement schedules of Carrols Restaurant Group, Inc. and subsidiary and the effectiveness of Carrols Restaurant Group, Inc. and subsidiary’s internal control over financial reporting appearing in this Annual Report on Form 10-K of Carrols Restaurant Group, Inc. for the year ended January 1, 2012.

/s/ Deloitte & Touche LLP

Rochester, New York

March 7, 2012