Exhibit 23.1

CONSENT OF THE INDEPENDENT AUDITORS

The Board of Directors

Impac Mortgage Holdings, Inc.:

We consent to the incorporation by reference in the Registration Statement on Form S-3 of Impac Mortgage Holdings, Inc., dated December 23, 2003, of our report dated January 29, 2003, except as to the last paragraph of Note M to the consolidated financial statements, which is as of February 27, 2003, relating to the consolidated balance sheets of Impac Mortgage Holdings, Inc. and subsidiaries as of December 31, 2002 and 2001, and the related consolidated statements of operations and comprehensive earnings (loss), changes in stockholders’ equity and cash flows for each of the years in the three-year period ended December 31, 2002, which report appears in the December 31, 2002, annual report on Form 10-K of Impac Mortgage Holdings, Inc. filed on March 28, 2003 and to the reference to our firm under the heading “Experts” in the prospectus and the prospectus supplement.

Our report refers to a change in the method of accounting for derivative instruments and hedging activities in 2001.

/s/ KPMG LLP

Los Angeles, California

December 23, 2003