EXHIBIT 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Impac Mortgage Holdings, Inc.:

We consent to the incorporation by reference in the registration statements (No. 333-12025, No. 333-68128, No. 333-83650, No. 333-106647, No. 333-117137 and No. 333-117070) on Form S-8 of Impac Mortgage Holdings, Inc. of our reports dated May 13, 2005, with respect to the consolidated balance sheets of Impac Mortgage Holdings, Inc. as of December 31, 2004 and 2003, and the related consolidated statements of operations and comprehensive earnings, changes in stockholders’ equity, and cash flows for each of the years in the three-year period ended December 31, 2004, management’s assessment of the effectiveness of internal control over financial reporting as of December 31, 2004 and the effectiveness of internal control over financial reporting as of December 31, 2004, which reports appear in the December 31, 2004 annual report on Form 10-K/A, Amendment No. 2 of Impac Mortgage Holdings, Inc.

Our report dated May 13, 2005, on management’s assessment of the effectiveness of internal control over financial reporting and the effectiveness of internal control over financial reporting as of December 31, 2004, expresses our opinion that Impac Mortgage Holdings, Inc. did not maintain effective internal control over financial reporting as of December 31, 2004 because of the effect of material weaknesses on the achievement of the objectives of the control criteria and contains an explanatory paragraph that states:

1) The Company’s internal controls intended to ensure the proper accounting and reporting for certain complex transactions and financial reporting matters were not designed or operating effectively as of December 31, 2004. For these purposes, complex transactions and financial reporting matters include those relating to the transfer of financial assets, derivative financial instruments, state income tax exposure items, and the income tax effect of intercompany transfers of financial assets between taxable and non-taxable operating segments. Specifically, the Company did not employ an adequate number of personnel in its accounting and finance departments with appropriate skills and expertise to ensure that the accounting and reporting for certain complex transactions and financial reporting matters included in the Company’s financial statements were in accordance with U.S. generally accepted accounting principles. As a result of these ineffective controls, the Company incorrectly recorded gains on sales of mortgage servicing rights when the related mortgage loans were sold to its parent company, the REIT. These gains on sales of mortgage servicing rights should have been recorded as an adjustment to the carrying value of the retained mortgage loans and

recognized as a yield adjustment over the remaining term of the loans. In addition, the Company did not identify certain loan purchase commitments as derivative financial instruments. Lastly, the Company did not prepare and maintain sufficient documentation of certain derivative financial instrument transactions to support hedge accounting. As a result, the Company did not reflect fluctuations in the estimated fair value of these derivative financial instruments in earnings in the period of change, as required by U.S. generally accepted accounting principles. The Company restated its financial statements in 2004 to correct these material errors in accounting for the years ended December 31, 2003, 2002 and 2001, and three months ended March 31, 2004 and 2003, the three and six months ended June 30, 2004 and 2003, and the three and nine months ended September 30, 2003.

2) The Company’s internal control over financial reporting intended to ensure adequate access and change control over end-user computing spreadsheets were not designed properly as of December 31, 2004. In addition, the information technology general controls related to access and program changes were deficient as of year end, resulting in a potential lack of reliability and integrity of the financial information which is used in these spreadsheets. As a result, although no actual misstatement was identified, there is a more than remote likelihood that financial statements and related footnote disclosures could be materially misstated. Specifically, there is the potential that an error could be reflected in the financial reporting and related disclosure of the allowance for loan losses, asset sales and securitizations and related yield adjustments on retained interests, and mortgage loan characteristics tables as a result of this material weakness in internal control over financial reporting.

/s/ KPMG LLP

Los Angeles, California

May 13, 2005