EXHIBIT 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

To the Board of Directors and Shareholders

United Security Bancshares

We consent to the incorporation by reference in Registration Statement No. 333-89362 dated May 30, 2002 on Form S-8, in Registration Statement No. 333-10078 dated September 25, 2002 on Form S-8, and in Registration Statement No. 333-134125 dated May 12, 2006 on Form S-8 of United Security Bancshares and Subsidiaries (United Security Bancshares) of our report dated March 21, 2014, with respect to the consolidated balance sheets of United Security Bancshares as of December 31, 2013 and 2012, and the related consolidated statements of income and comprehensive income, changes in shareholders’ equity and cash flows for each of the years in the two year period ended December 31, 2013, which report is included in this annual report on Form 10-K of United Security Bancshares for the year ended December 31, 2013.

/s/ Moss Adams, LLP

Stockton, California

March 21, 2014