Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the reference to our firm under the caption “Experts” in the Amendment No. 1 to the Registration Statement (Form S-4 No. 333-133665) of Capital One Financial Corporation and the related Prospectus of Capital One Financial Corporation for the registration of common stock of Capital One Financial Corporation and the related Proxy Statements of Capital One Financial Corporation and North Fork Bancorporation, Inc. for the proposed merger of Capital One Financial Corporation and North Fork Bancorporation, Inc. and to the incorporation by reference therein of our report dated February 15, 2006, except for Note 27, as to which the date is February 27, 2006, with respect to the consolidated financial statements of Capital One Financial Corporation, and our report dated February 15, 2006, with respect to Capital One Financial Corporation management’s assessment of the effectiveness of internal control over financial reporting, and the effectiveness of internal control over financial reporting of Capital One Financial Corporation, included in its Annual Report (Form 10-K) for the year ended December 31, 2005, filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

McLean, Virginia

June 5, 2006