Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Impac Mortgage Holdings, Inc.:

We consent to the incorporation by reference in the registration statements (No. 333-12025, No. 333-68128, No. 333-83650, No. 333-106647, No. 333-117137, and No. 333-117070) on Form S-8 and registration statement (No. 333-111517) on Form S-3 of Impac Mortgage Holdings, Inc., of our report dated January 29, 2004, except as to Note T to the consolidated financial statements, which is as of February 15, 2004, and Note A.2, which is as of August 16, 2004, relating to the consolidated balance sheets of Impac Mortgage Holdings, Inc. and subsidiaries as of December 31, 2003 and 2002, and the related consolidated statements of operations and comprehensive earnings, changes in stockholders’ equity and cash flows for each of the years in the three-year period ended December 31, 2003, which report appears in the December 31, 2003, annual report on Form 10-K/A of Impac Mortgage Holdings, Inc.

As discussed in Note A.16 to the consolidated financial statements, the Company changed its method of accounting for derivative instruments and hedging activities in 2001. Additionally, as discussed in Note A.2, the Company has restated the consolidated financial statements as of December 31, 2003 and 2002, and for each of the years in the three-year period ended December 31, 2003.

/s/ KPMG LLP

Orange County, California

October 13, 2004