Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statements (Form S-3 Nos. 333-118966, 333-112598 and 333-90742 and Form S-8 Nos. 333-112561, 333-112218, 333-136890, 33-52892, 333-164618, 333-163245, 333-170708, and 333-170764) of Destination XL Group, Inc. of our reports dated March 17, 2014, with respect to the consolidated financial statements of Destination XL Group, Inc. and the effectiveness of internal control over financial reporting of Destination XL Group, Inc., included in the Annual Report (Form 10-K) for the year ended February 1, 2014.

/s/ KPMG LLP

Boston, Massachusetts

March 17, 2014