EXHIBIT 23

Consent of Independent Registered Public Accounting Firm

The Board of Directors and Stockholders

Heritage Financial Corporation:

We consent to the incorporation by reference in the registration statements (No. 333-57513, No. 333-71415, No. 333-87599, No.333-88976, No.333-88980, No.333-88982, No.333-134473, No.333-134474, No.333-134475 and No. 333-167146) on Form S-8 and the registration statements (No. 333-156271 and No. 333-167145) on Form S-3 of Heritage Financial Corporation of our reports dated March 2, 2012, with respect to the consolidated statements of financial condition of Heritage Financial Corporation and subsidiaries as of December 31, 2011 and 2010, and the related consolidated statements of operations, stockholders’ equity and comprehensive income (loss) and cash flows for each of the years in the three-year period ended December 31, 2011, and the effectiveness of internal control over financial reporting as of December 31, 2011, which reports appear in the December 31, 2011 annual report on Form 10-K of Heritage Financial Corporation.

/s/ KPMG, LLP

Seattle, Washington

March 2, 2012