Exhibit 23.4

Consent of Independent Registered Public Accounting Firm

We consent to the reference to our firm under the caption “Experts” and to the use of our report dated February 27, 2012, with respect to the balance sheets of 1st Security Bank of Washington as of December 31, 2011 and 2010, and the related statements of income, equity and comprehensive income, and cash flows for the years then ended, which is included in Amendment No. 2 to the Registration Statement (Form S-1, No. 333-177125) of FS Bancorp, Inc. and related Prospectus for the registration of between 2,082,500 and 3,240,125 shares of common stock.

/s/ Moss Adams LLP

Bellingham, Washington

May 2, 2012