Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Jos. A. Bank Clothiers, Inc.:

We consent to the incorporation by reference in Registration Statement Nos. 333-103962, 333-85426, 333-57492 and 333-20363 on Forms S-8 of our reports dated April 13, 2005, relating to the financial statements of Jos. A. Bank Clothiers, Inc. and management’s report on the effectiveness of Internal Controls for Financial Reporting (which report expresses an unqualified opinion on management’s assessment that the Company’s internal control over financial reporting is not effective and expresses an adverse opinion on the effectiveness of the Company’s internal control over financial reporting because of a material weakness), appearing in the Annual Report on Form 10-K of Jos. A. Bank Clothiers, Inc. for the year ended January 29, 2005.

/s/ Deloitte & Touche LLP

Baltimore, Maryland

April 13, 2005

1