Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

To the Board of Directors

United Security Bancshares and Subsidiary

We consent to the incorporation by reference in Registration Statement No. 333-89362 dated May 30, 2002 on Form S-8, in Registration Statement No. 333-10078 dated September 25, 2002 on Form S-8, and in Registration Statement No. 333-134125 dated May 12, 2006 on Form S-8 of United Security Bancshares and Subsidiary (United Security Bancshares) of our report dated March 17, 2015, with respect to the consolidated balance sheets of United Security Bancshares as of December 31, 2014 and 2013, and the related consolidated statements of income, comprehensive income, shareholders’ equity and cash flows for each of the years in the two year period ended December 31, 2014, which is included in this annual report on Form 10-K of United Security Bancshares for the year ended December 31, 2014.

/s/ Moss Adams LLP

Stockton, California

March 17, 2015