Exhibit 23

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statements (Form S-8 No. 333-168839, No. 333-170579, No. 333-218723, and No. 333-237039) of Territorial Bancorp Inc. of our report dated March 21, 2022, with respect to the consolidated financial statements of Territorial Bancorp Inc. and Subsidiaries, appearing in this Annual Report on Form 10-K for the year ended December 31, 2021.

/s/ Moss Adams LLP

Portland, Oregon

March 21, 2022