QuickLinks -- Click here to rapidly navigate through this document

Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

           We consent to the incorporation by reference in the Registration Statements on Form S-3 (No. 333-121562) and Form S-8 (Nos. 333-12025, 333-68128, 333-83650, 333-106647, 333-117070, 333-117137 and 333-128113) and related Prospectuses of Impac Mortgage Holding, Inc. of our reports dated March 7, 2006 with respect to the consolidated financial statements of Impac Mortgage Holding, Inc., management's assessment of the effectiveness of internal control over financial reporting, and the effectiveness of internal control over financial reporting of Impac Mortgage Holding, Inc., in this Annual Report (Form 10-K) for the year ended December 31, 2005.

/s/  ERNST & YOUNG LLP

Los Angeles, California
March 10, 2006

QuickLinks

Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM