Exhibit 23

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statements No. 333-168839, No. 333-170579, No. 333-218723 and No. 333-237039 on Form S-8 of Territorial Bancorp Inc. of our reports dated March 19, 2021, with respect to the consolidated financial statements of Territorial Bancorp Inc. and Subsidiaries and the effectiveness of internal control over financial reporting, appearing in this Annual Report on Form 10-K for the year ended December 31, 2020.

/s/ Moss Adams LLP

Portland, Oregon

March 19, 2021