Exhibit 23.1

Independent Auditors’ Consent

The Board of Directors

Charter Financial Corporation:

We consent to the incorporation by reference in the registration statements (No. 333-67402 and No. 333-98161) on Form S-8 of Charter Financial Corporation and subsidiaries of our report dated November 14, 2003, with respect to the consolidated balance sheets of Charter Financial Corporation and subsidiaries as of September 30, 2003 and 2002, and the related consolidated statements of income, stockholders’ equity and comprehensive income (loss), and cash flows, for each of the years in the three-year period ended September 30, 2003 which report appears in the September 30, 2003, annual report on Form 10-K of Charter Financial Corporation.

/s/ KPMG LLP

Atlanta, Georgia

December 22, 2003