Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Charter Financial Corporation:

We consent to the incorporation by reference in Registration Statements (No. 333-67402, No. 333-98161 and No. 333-104085) all on Forms S-8 of Charter Financial Corporation and subsidiaries of our report dated December 7, 2006, with respect to (i) consolidated balance sheets of Charter Financial Corporation and subsidiaries as of September 30, 2006 and 2005, and the related consolidated statements of income, stockholders’ equity and comprehensive income (loss), and cash flows, for each of the years in the three-year period ended September 30, 2006; (ii) management’s assessment of the effectiveness of internal control over financial reporting as of September 30, 2006; and (iii) the effectiveness of internal control over financial reporting as of September 30, 2006; which reports appear in the September 30, 2006 annual report on Form 10-K of Charter Financial Corporation.

As discussed in Note 1 to the consolidated financial statements, the Company adopted the provisions of Statement of Financial Accounting Standards No. 123R, Share-Based Payments, effective October 1, 2005.

/s/ KPMG LLP

Birmingham, Alabama

December 7, 2006