QuickLinks -- Click here to rapidly navigate through this document

Exhibit 23

Independent Auditors' Consent

The Board of Directors
Jos. A. Bank Clothiers, Inc.:

        We consent to the incorporation by reference in the Registration Statement on Form S-8 (File Nos. 333-103962, 333-85426, 333-57492 and 333-20363) of Jos. A. Bank Clothiers, Inc. of our report dated March 25, 2004, with respect to the consolidated balance sheets of Jos. A. Bank Clothiers, Inc. and subsidiaries as of February 1, 2003 and January 31, 2004, and the related consolidated statements of income, stockholders' equity, and cash flows for the years then ended, which report appears in the January 31, 2004 annual report on Form 10-K of Jos. A. Bank Clothiers, Inc.

/s/ KPMG LLP
Baltimore, Maryland
April 14, 2004

QuickLinks

Exhibit 23

Independent Auditors' Consent