EXHIBIT 10.22

UNITED SECURITY BANCSHARES, INC.

NON-EMPLOYEE DIRECTORS’ DEFERRED COMPENSATION PLAN

The Board of Directors of United Security Bancshares, Inc. hereby establishes the United Security Bancshares, Inc. Non-Employee Directors’ Deferred Compensation Plan, to be effective as of the 22nd day of January, 2004.

ARTICLE I

PURPOSE OF PLAN

The purpose of the Plan is to assist the Directors of the Holding Company and the Bank in planning for their retirement.

ARTICLE II

DEFINITIONS

Where the following terms appear herein, they shall have the respective meanings set forth in this ARTICLE II, unless the context clearly indicates to the contrary.

Section 2.1. “Bank” shall mean First United Security Bank, a wholly-owned subsidiary of the Holding Company.

Section 2.2. “Board of Directors” shall mean the board of directors of the Holding Company.

Section 2.3. “Committee” shall mean the compensation committee of the board of directors of the Holding Company.

Section 2.4. “Holding Company” shall mean United Security Bancshares, Inc.

Section 2.5. “Compensation” shall mean a Participant’s compensation for services as a Director.

Section 2.6. “Deferral Termination Date” shall mean the date a Participant ceases to be a Director.

Section 2.7. “Deferred Compensation” shall mean Compensation deferred pursuant to the provisions of the Plan.

Section 2.8. “Deferred Compensation Account” shall mean the Participant’s account established pursuant to Section 4.3 herein.

Section 2.9. “Director” shall mean a non-employee member of the board of directors of the Holding Company or the board of directors of the Bank.

Section 2.10. “Market Price” shall mean the closing sales price per share for the common stock of the Holding Company or, if no closing sales price is reported, the average of the bid and ask prices per share on such date, as reported on the The Nasdaq SmallCap Market.

Section 2.11. “Participant” shall mean any non-employee Director of the Holding Company or the Bank who has elected to have all or a part of his or her Compensation deferred pursuant to the Plan.

Section 2.12. “Plan” shall mean this United Security Bancshares, Inc. Non-Employee Directors’ Deferred Compensation Plan, as the same may hereafter be amended from time to time.

Section 2.13. “Plan Earnings” shall mean amounts of interest to which reference is made in Section 5.1 herein and of dividends and distributions to which reference is made in Sections 6.2 and 6.3 herein.

Section 2.14. “Stock Equivalent” shall mean a measure of value equal to one share of the Holding Company’s common stock and shall not include stock appreciation rights.

Section 2.15. “Stock Equivalents Account” shall mean the Participant’s account established pursuant to Section 4.3 herein.

ARTICLE III

ADMINISTRATION OF THE PLAN

Section 3.1. Committee. The Committee or such successor Committee as may be duly appointed by the Board of Directors shall administer, construe and interpret the Plan. Decisions of the Committee with respect to any matter involving the Plan shall be final and binding on the Holding Company, the Bank and all Participants. The Committee shall maintain complete and adequate records pertaining to the Plan, including but not limited to Deferred Compensation Accounts and Stock Equivalent Accounts.

Section 3.2. Indemnity.

(a) The Holding Company (the “Indemnifying Party”) hereby agrees to indemnify and hold harmless the members of the Committee (the “Indemnified Parties”) against any losses, claims, damages or liabilities to which any of the Indemnified Parties may become subject to the extent that such losses, claims, damages, liabilities or actions in respect thereof arise out of or are based upon any act or omission of such Indemnified Party in connection with the administration of the Plan (including any act or omission constituting negligence on the part of such Indemnified Party, but excluding any act or omission constituting gross negligence or willful misconduct on the part of such Indemnified Party), and will reimburse the Indemnified Party for any legal or other expenses reasonably incurred by him or her in connection with investigating or defending against any such loss, claim, damage, liability or action.

(b) Promptly after receipt by the Indemnified Party under Section 3.2(a) herein of notice of the commencement of any action or proceeding for which the Indemnified Party believes he or she is indemnified under Section 3.2(a), the Indemnified Party shall notify the Indemnifying Party in writing of the commencement thereof; provided, however, that the failure to notify the Indemnifying Party shall not relieve it from any liability which it may have to the Indemnified Party to the extent the Indemnifying Party is not prejudiced by such failure. If any such action or proceeding shall be brought against the Indemnified Party and it shall notify the Indemnifying Party of the commencement thereof, the Indemnifying Party shall be entitled to participate therein, and, to the extent that it shall wish, to assume the defense thereof, with counsel reasonably satisfactory to the Indemnified Party, and the Indemnifying Party shall not be liable to such Indemnified Party under Section 3.2(a) for any legal or other expenses subsequently incurred by the Indemnified Party in connection with the defense thereof, other than reasonable costs of investigation or reasonable expenses of actions taken at the written request of the

Indemnifying Party. The Indemnifying Party shall not be liable for any compromise or settlement of any such action or proceeding effected without its consent, which consent will not be unreasonably withheld.

ARTICLE IV

DEFERRED COMPENSATION

Section 4.1. Initial Elections by Directors. Any Director may (1) within thirty (30) days of the later of (a) the date the Director first becomes a Director or (b) the date the Plan becomes effective or (2) within thirty (30) days prior to a calendar year, elect to participate in the Plan and to have all or a portion of the Compensation otherwise payable to him or her as a Director deferred and paid to him or her after his or her Deferral Termination Date. In making such initial election hereunder, a Director shall specify (i) the portion, if any, of such Deferred Compensation which will be (A) held subject to the interest payment provisions of ARTICLE V hereof or (B) translated into Stock Equivalents in accordance with ARTICLE VI hereof, and (ii) the form of payment of the Deferred Compensation: (A) a single lump-sum payment, (B) five (5) annual installments (i.e., one-fifth of the account in year one, one-fourth of the account in year two, etc.) or (C) ten (10) annual installments (i.e., one-tenth of the account in year one, one-ninth of the account in year two, etc.). Such election shall be made in a writing delivered to the Committee and shall be applicable only with respect to Compensation earned after the end of the calendar month in which such election is made.

Section 4.2. Subsequent Elections by Directors.

(a) Within thirty (30) days prior to the beginning of any calendar year following a Director’s initial election described in Section 4.1, a Participant may make a subsequent election to increase or decrease the portion of his or her Compensation to be deferred pursuant to the Plan and to elect the portion of such Deferred Compensation and any Plan Earnings to be (i) held subject to the interest payment provisions of ARTICLE V hereof or (ii) translated into Stock Equivalents in accordance with ARTICLE VI hereof. Any such election shall be effective as of the first day of the calendar year following the calendar year in which such election is made. Notwithstanding anything to the contrary herein, no such subsequent election shall affect a transfer of any amount credited, as of the first day of such calendar year, to either the Deferred Compensation Account or the Stock Equivalents Account from such account to the other account.

(b) Subsequent to the initial election by a Participant described in Section 4.1 and at least twelve (12) months prior to a Participant’s Deferral Termination Date, the Committee, in its sole discretion, may authorize a Participant to extend the form of payment period beyond that originally elected by the Participant pursuant to Section 4.1 so long as the payment period does not exceed the form of payment methods described in Section 4.1. Such subsequent election to extend the form of payment period beyond that originally elected by the Participant shall not become effective until twelve (12) months after the date such election is extended by the Committee (“12-Month Waiting Period”). If a Participant’s Deferral Termination Date occurs prior to the expiration of the 12-Month Waiting Period, any payments made to the Participant during the 12-Month Waiting Period shall be paid in accordance with the Participant’s initial form of payment election made pursuant to Section 4.1.

Section 4.3. Establishment of Deferred Compensation Accounts and Stock Equivalents Accounts. There shall be established for each Participant an account to be designated as such Participant’s Deferred Compensation Account and an account to be designated as such Participant’s Stock Equivalents Account, as appropriate.

Section 4.4. Allocations to Accounts. Any Deferred Compensation earned by a Participant during a calendar month shall be credited to the Participant’s Deferred Compensation Account as of the end of the calendar month. Plan Earnings that consist of interest accrued on a Participant’s Deferred Compensation Account due to the portion of Deferred Compensation that is subject to the interest payment provisions of ARTICLE V shall be credited to such Participant’s Deferred Compensation Account as of the end of each month. Plan Earnings that consist of dividends (cash or otherwise) and distributions accrued on a Participant’s Stock Equivalents Account due to the portion of Deferred Compensation that has been translated into Stock Equivalents in accordance with ARTICLE VI shall be credited to such Participant’s Deferred Compensation Account as of the end of each calendar quarter prior to being translated into Stock Equivalents in accordance with ARTICLE VI.

ARTICLE V

DEFERRED COMPENSATION SUBJECT TO INTEREST

Section 5.1. Interest on Deferred Compensation Accounts. A Participant’s Deferred Compensation Account shall be credited as of the end of each calendar month with an amount equivalent to interest for the number of days in such calendar month at the 30-Day London Interbank Offered Rate (LIBOR) plus 75 basis points for the last business day of the immediately preceding calendar month as reported on the Bloomberg financial news system applied to the balance of such Deferred Compensation Account (excluding the portion of the Deferred Compensation Account that will be translated into Stock Equivalents in accordance with ARTICLE VI) at the beginning of such calendar month; provided, however, no amount credited to a Participant’s Deferred Compensation Account subsequent to the beginning of a calendar month shall bear interest during that calendar month. Interest credited to a Participant’s Deferred Compensation Account shall be held in such account subject to the provisions of Section 4.4 herein.

Section 5.2. Distribution of Deferred Compensation Accounts Subject to Interest.

(a) When a Participant’s Deferral Termination Date shall occur, the balance in such Participant’s Deferred Compensation Account shall be distributed in cash to such Participant in the form elected by the Participant pursuant to Section 4.1. Until payment is made, interest shall continue to accrue in the manner provided in Section 5.1. Plan Earnings accrued to the date of payment of any lump-sum or annual installment on such amount shall be paid in conjunction with such payment. The lump-sum or initial annual installment shall be distributed within thirty (30) days after a Participant’s Deferral Termination Date. The remaining installments, if appropriate, shall be distributed at annual intervals thereafter.

(b) If a Participant’s Deferral Termination Date shall occur by reason of his or her death or if he or she shall die after his or her Deferral Termination Date, but prior to receipt of all payments provided for in this Section, all payments distributable hereunder shall be distributed in a lump-sum to such Participant’s beneficiary designated on his or her election form, or if none is designated, to such Participant’s estate or personal representative, as soon as administratively feasible following his or her death.

ARTICLE VI

STOCK EQUIVALENTS

Section 6.1. Stock Equivalents Accounts. The number of Stock Equivalents, or fractions thereof, to be credited to a Participant’s Stock Equivalents Account in accordance with Section 4.4 shall be determined by dividing the amount of Deferred Compensation and Plan Earnings to be allocated to

such account pursuant to the Participant’s specifications given in accordance with ARTICLE IV by the Market Price on the last trading day of the calendar quarter specified in Section 4.4. The number of Stock Equivalents, so determined, shall be credited to the Stock Equivalents Account established for the Participant.

Section 6.2. Cash and Property Dividend Credits. Additional credits shall be made to a Participant’s Deferred Compensation Account throughout the period of such Participant’s participation in the Plan, and thereafter until all distributions to which the Participant is entitled under Section 6.5 or ARTICLE VIII shall have been made, in amounts equal to the Plan Earnings consisting of the cash or fair market value of any dividends or distributions declared and made with respect to the Holding Company’s common stock payable in cash, securities issued by the Holding Company (other than the Holding Company’s common stock but including any such securities convertible into the Holding Company’s common stock) or other property which the Participant would have received had he or she been the owner on the record dates for the payment of such dividends of the number of shares of the Holding Company’s common stock equal to the number of Stock Equivalents in his or her Stock Equivalents Account on such dates. Each such credit shall be effected as of the end of the calendar quarter in which such dividend or distribution is made. Each and every amount so credited to a Participant’s Deferred Compensation Account shall be held in such account subject to the provisions of Section 4.4 herein.

Section 6.3. Stock Dividend Credits. Additional credits shall be made to a Participant’s Stock Equivalents Account throughout the period of his or her participation in the Plan, and thereafter until all distributions to which the Participant is entitled under Section 6.5 or ARTICLE VIII shall have been made, of a number of Stock Equivalents equal to the number of shares (including fractional shares) of the Holding Company’s common stock to which the Participant would have been entitled as common stock dividends had such Participant been the owner on the record dates for the payments of such stock dividends of the number of shares of the Holding Company’s common stock equal to the number of Stock Equivalents credited to his or her Stock Equivalents Account on such dates. Such additional credits shall be effected as of the end of the calendar quarter in which payment of such stock dividend is made.

Section 6.4. Recapitalization. If, as a result of a split or combination of the Holding Company’s outstanding common stock or other recapitalization or reorganization, the number of shares of the Holding Company’s outstanding common stock is increased or decreased or all or a portion of the Holding Company’s outstanding common stock is exchanged for or converted into other securities issued by the Holding Company (including without limitation securities convertible into the Holding Company’s common stock) or other property, the number of Stock Equivalents credited to a Participant’s Stock Equivalents Account shall, to the extent reasonably practicable, be equitably adjusted to give effect to such recapitalization or reorganization as if the Participant had owned of record on the effective date of such recapitalization or reorganization a number of shares of the Holding Company’s common stock equal to the number of Stock Equivalents credited to his or her Stock Equivalents Account immediately prior thereto.

Section 6.5. Distributions from Stock Equivalent Account. When a Participant’s Deferral Termination Date shall occur, the Holding Company shall become obligated to make the distributions prescribed in paragraphs (a) and (b) below. At the time of any distribution, each Stock Equivalent to be distributed shall be converted into one share of the Holding Company’s common stock and such share shall be distributed to the Participant. Any fraction of a Stock Equivalent to be distributed shall be converted into an amount in cash equal to the Market Price of one share of the Holding Company’s common stock on the trading day next preceding the date of distribution multiplied by such fraction and such cash shall be distributed to the Participant. Additionally, any amounts earned by a Participant

during a calendar quarter not yet allocated to the Participant’s Stock Equivalents Account in accordance with Section 6.1 herein shall be distributed to the Participant in cash.

(a) Distribution shall be made in the form elected by the Participant pursuant to Section 4.1. Until payment is made, Plan Earnings shall continue to be credited in the manner provided in Sections 6.2 and 6.3 herein. Plan Earnings accrued to the date of payment of any lump-sum or annual installment on such amount shall be paid in conjunction with such payment. The lump-sum or initial annual installment shall be distributed within thirty (30) days after a Participant’s Deferral Termination Date. The remaining installments, if appropriate, shall be distributed at annual intervals thereafter.

(b) If a Participant’s Deferral Termination Date shall occur by reason of his or her death or if he or she shall die after his or her Deferral Termination Date but prior to receipt of all distributions provided for in this Section, all Stock Equivalents, or the undistributed balance thereof, shall be distributed to such Participant’s beneficiary designated on his or her election form, or if none is designated, to such Participant’s estate or personal representative, as soon as administratively feasible following his or her death.

ARTICLE VII

NATURE OF PLAN

Neither the Holding Company nor the Bank is under any obligation (1) to transfer amounts credited to a Participant’s Deferred Compensation Account or Stock Equivalents Account to any trust or escrow account or (2) to secure any amount credited to a Participant’s Deferred Compensation Account or Stock Equivalents Account by any specific assets of the Holding Company or the Bank or any other asset in which the Holding Company or the Bank has an interest. The Plan shall not be construed to require the Holding Company or the Bank to fund any of the benefits provided hereunder nor to establish a trust for such purpose. The Holding Company or the Bank may make such arrangements as it desires to provide for the payment of benefits, including, but not limited to, the establishment of a rabbi trust or such other equivalent arrangement as the Holding Company or the Bank may decide. No such arrangement shall cause the Plan to be a “funded” plan within the meaning of Title I of the Employee Retirement Income Security Act of 1974, as amended, nor shall any such arrangement change the nature of the obligation of the Holding Company or the Bank or the rights of the Participants as provided herein. Neither a Participant nor his or her beneficiary, estate or personal representative shall have any rights against the Holding Company or the Bank with respect to any portion of a Deferred Compensation Account or Stock Equivalents Account or any trust, escrow account or other arrangement established in connection with the Plan except as a general unsecured creditor.

ARTICLE VIII

TERMINATION OF THE PLAN

The Board of Directors may terminate the Plan at any time. Upon termination of the Plan, distributions in respect of credits to Participants’ Deferred Compensation Accounts and Stock Equivalents Accounts as of the date of termination shall be made in the manner and at the time prescribed in Section 5.2 or Section 6.5; provided, however, that the Board of Directors shall have the right to cause distributions in respect of credits to Participants’ Deferred Compensation Accounts and Stock Equivalents Accounts as of the effective date of such termination of the Plan to be made at such time and in such manner as it may determine; and further provided that the value of the accounts on distribution shall be determined in a manner consistent with the provisions of Section 5.2 and Section 6.5, as applicable.

ARTICLE IX

AMENDMENT OF THE PLAN

The Board of Directors may, without the consent of Participants or their beneficiaries, amend the Plan at any time and from time to time; provided, however, that no amendment may deprive a Participant of the amounts allocated to his or her Deferred Compensation Account or Stock Equivalents Account or be retroactive in effect to the prejudice of any Participant.

ARTICLE X

GENERAL PROVISIONS

Section 10.1. No Preference. No Participant shall have any preference over the general creditors of the Holding Company or the Bank in the event of the Holding Company’s or the Bank’s insolvency.

Section 10.2. Authorized Payments. Notwithstanding any other provisions of the Plan, if any amounts payable under the Plan are found in a “determination” (within the meaning of Section 1313(a) of the Internal Revenue Code of 1986, as amended) to have been includible in the gross income of a Participant prior to payment of such amounts hereunder, such amounts shall be paid to such Participant as soon as practicable after the Committee is advised of such determination. For purposes of this Section, the Committee shall be entitled to rely on an affidavit by a Participant and a copy of the determination to the effect that a determination described herein has occurred.

Section 10.3. Gender Words. Wherever any words are used herein in the masculine, feminine or neuter gender, they shall be construed as though they were also used in another gender in all cases where they would so apply, and whenever any words are used herein in the singular or plural form, they shall be construed as though they were also used in the other form in all cases where they would so apply.

Section 10.4. Assignment of Benefits. Benefits provided under the Plan may not be assigned or alienated, either voluntarily or involuntarily, other than by will or the applicable laws of descent and distribution; provided, however, in the event the Participant is indebted to the Holding Company or the Bank, such benefits may be used to offset such indebtedness prior to distribution.

Section 10.5. Conflicts of Laws. The laws of the State of Alabama shall govern the interpretation and performance of the terms of the Plan. The Plan is not intended to qualify under Section 401(a) of the Internal Revenue Code of 1986, as amended, or to comply with the Employee Retirement Income Security Act of 1974, as amended.