EXHIBIT 23

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Heritage Financial Corporation:

We consent to incorporation by reference in the registration statements (No. 333-57513, No. 333-71415, No. 333-87599, No. 333-88980, No. 333-88982 and No. 333-88976) on Form S-8 of Heritage Financial Corporation of our report dated March 15, 2005, with respect to the consolidated statements of financial condition of Heritage Financial Corporation and subsidiaries as of December 31, 2004 and 2003, and the related consolidated statements of income, stockholders’ equity and comprehensive income, and cash flows for each of the years in the three-year period ended December 31, 2004, management’s assessment of the effectiveness of internal control over financial reporting as of December 31, 2004 and the effectiveness of internal control over financial reporting as of December 31, 2004, which reports appear in the December 31, 2004 Annual Report on Form 10-K of Heritage Financial Corporation.

/s/ KPMG LLP

Seattle, Washington

March 15, 2005