Exhibit 23.4

CONSENT OF THE INDEPENDENT AUDITORS

The Board of Directors

Impac Mortgage Holdings, Inc.:

We consent to the incorporation by reference in Post Effective Amendment No. 1 to the Registration Statement (No. 333-74432) on Form S-3 of Impac Mortgage Holdings, Inc., of our report dated January 28, 2002, except as to note T to the consolidated financial statements, which is as of March 8, 2002, relating to the consolidated balance sheets of Impac Mortgage Holdings, Inc. and subsidiaries as of December 31, 2001 and 2000, and the related consolidated statements of operations and comprehensive earnings (loss), changes in stockholders’ equity and cash flows for each of the years in the three-year period ended December 31, 2001, which report appears in the December 31, 2001, annual report on Form 10-K of Impac Mortgage Holdings, Inc. filed on April 1, 2002 and to the reference to our firm under the heading “Experts” in the prospectus and the prospectus supplement.

Our report refers to a change in the method of accounting for derivative instruments and hedging activities in 2001.

/s/ KMPG LLP

Los Angeles, California

February 20, 2003