Comment Letter Process

This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.

Comment Letter Process

Audit Report No. 259
February 2, 1998

INDEX

Executive Summary
Scope And Objectives
Audit Methodology
Background
Audit Results
    Overall Results
    Communicate Staff Findings
    Management Review
    Communicate Comments
    Resolve Comments
    Program Performance Measures
    Staff Performance Measures

Appendix A - Control Self Assessment Methodology
    Management Objectives
    The Workshops
    Data Evaluation
        Quantitative Analysis
        Qualitative Analysis
    Audit Report

Appendix B - Effectiveness Rating Scales
    Actual Effectiveness
    Importance
 

Executive Summary

The Securities and Exchange Commission's Office of Inspector General (Office) evaluated the Division of Corporation Finance's (Division) comment letter process. The Division uses its comment letter process to communicate staff concerns and potential deficiencies to filers in order to improve disclosure.

We obtained and analyzed information concerning successes, obstacles, recommendations, and effectiveness ratings related to the primary objective and six supporting objectives identified by the Division for the comment letter process. To obtain this information, we conducted six internal audit workshops involving approximately seventeen per cent of Division staff. Separately, we also reviewed feedback from issuers on the process and analyzed information on comment letters and filing processing to determine conformance to Division procedures and timeliness goals.

Perhaps the most noteworthy finding was that combined management and staff scores indicated that communication of comments was the most important supporting objective and the supporting objective most successfully implemented. Management and staff also agreed that the Division is composed of, for the most part, hardworking, conscientious, highly qualified and motivated staff who are dedicated to improving disclosure to investors. Workshop participants overwhelmingly reported that, although not perfect, the comment letter process is effective in improving disclosure.

The participants in the workshops expressed a desire for better communication of Division filing review policies from management to staff; more consistent treatment of filing issues; clearer definition of accountant, examiner, and reviewer responsibilities; and re-evaluation of the filing examination report format. The Office generally endorses these recommendations and makes additional recommendations in its report. These recommendations include updating review guidance as appropriate, designing and implementing formal training for reviewers, and improving the use of outcome-based program performance measures.

Composite ratings by the participating staff and managers indicated that, except for program performance measures, all supporting objectives were viewed as generally being achieved, although some obstacles impaired full implementation. Both managers' and staff's scores indicated that the program performance measure objective was generally not achieved. The staff and management disagreed about whether the measurement of staff performance was achieving the Division's objectives. The staff thought the objective was not being achieved; management rated it as generally being achieved although some obstacles impaired full implementation. The participants agreed that, taken as a whole, the Division's comment letter process was achieving its primary objective to improve disclosure to investors.

The Division gave us information on issuers that were contacted by the Chairman within the past year for their views on the filing review process. After reviewing the information we contacted five companies not on the Division's list for their feedback on the Division's comment letter process. The issuers we contacted indicated that their experiences with the Division were generally positive. Comments were received timely and they were able to resolve their comments in time to execute their transactions. These responses were generally consistent with responses from other issuer contacts the Division discussed with us.

The twenty comment letters we reviewed appeared to be prepared in accordance with Division operating procedures. We also reviewed a list of 1933 Act registrations filed between October 1, 1996 and June 31, 1997 to determine the timeliness of filing review comments. Our review indicated that the Division issued comments within its 30 day timeframe on approximately 88% of the 1933 Act filings in our sample receiving full reviews and approximately 98% of the 1933 Act filings receiving a monitor for one or more specific items.
 

SCOPE AND OBJECTIVES

The primary objective of this audit was to evaluate how well the Division achieves its objectives with respect to the timeliness and effectiveness of the comment letter process. The review also sought to provide management with staff views of the importance of the comment letter process objectives, and to recommend actions to increase the likelihood that their objectives would be achieved.

During the audit, management designated the primary objective of the comment letter process to be:

Improve Disclosure to Investors

Division management also identified six supporting objectives. The supporting objectives reflect activities that increase the chances of achieving the primary objective. They were:

Communicate Staff Findings - Document and communicate to supervisors staff findings from the review of disclosure filings by preparing an examination report/draft comments.

Management Review - Review the examination report/draft comments for policy considerations (e.g., materiality, uniformity).

Communicate Comments - Develop and communicate (written and verbal) comments to the issuers or their representatives.

Resolve Comments - Gain issuer acceptance of comments through implementing amendments and futures comments.

Program Performance Measures - Measure and evaluate the performance of the comment letter process accurately.

Staff Performance Measures - Evaluate staff performance in the comment letter process to improve individual and program performance.

The field work was performed between May and October 1997 in accordance with generally accepted government auditing standards.
 

AUDIT METHODOLOGY

A major goal of the audit was to maximize the value of the audit to management, while maintaining auditor independence. A version of a private sector, internal audit methodology (Control Self-Assessment or CSA) was adapted for this purpose. The Institute of Internal Auditors has promoted the concept internationally for the last several years with outstanding results reported.

The premises for using this approach are common to many applications that use a team approach to decision making (e.g., teams can process more and more diverse information than an individual, employees are closer to the information needed than anyone else, staff can make a good system fail or vice versa, etc.). The CSA methodology used in the audit consists of four primary tasks: identify management objectives, convene workshops to discuss and rate each objective, evaluate the workshop data, and prepare an audit report. These audit steps are described in further detail in Appendix A.

We collected data primarily through six workshops (fifty-six staff or approximately 17% of Division personnel participated in the workshops). The workshop participants discussed and anonymously rated each supporting objective. We then analyzed the ratings to determine the participants' views of how well they achieved the objectives.

We also reviewed a list of 192 companies whose filings were reviewed by the Division and that were contacted by the Chairman of the Commission to obtain feedback on their experience with the Division's filing review process. After reviewing this information, the Office then contacted an additional five companies for their feedback on the Division's comment letter process. We also reviewed reports of registrations filed between October 1, 1996 and June 31, 1997 (1,228 registrations receiving full reviews and 567 registrations receiving a monitor for one or more specific items) to determine the timeliness of the initial filing review comments. In addition, we reviewed twenty comment letters to determine if they were prepared in accordance with Division operating procedures.
 

BACKGROUND

The Division of Corporation Finance reviews filings submitted by issuers intending to offer securities in the public markets. The filings are received by the Commission, sent to the Division, and then screened for review. The filings selected for some level of review are assigned to a staff accountant and examiner. The staff accountant performs the initial review of the financial statements in the filing for compliance with Generally Accepted Accounting Principles and other accounting rules and regulations. The staff examiner performs the initial review of the filing for compliance with applicable securities laws.

During the filing review, the accountant and examiner maintain contact with the issuer for clarifications and additional information relating to the disclosures in the filing. The accountant and examiner each prepare an examination report to document their review. Also, they list any deficiencies in the filing's disclosures and propose comments relating to those deficiencies.

After completing the review, designated senior staff (reviewers) look over the filing and proposed comments. As a result of their review, they edit and add comments as appropriate. The comments are then faxed to the issuer.

Comments may consist of requests for supplemental information, amendments to filings, or improvements in future filings (futures comments). Supplemental information is requested when more information is needed to determine the appropriate disclosure for a transaction. Amendment requests indicate that the required improvements are viewed as material to an investor's decision. Staff requests futures comments when the improvements are not immediately consequential, but will improve disclosure.

The Division works with the issuer to resolve the comments. Initially, the issuer communicates with the staff accountant and examiner. However, the issuer may also request consideration of a question by more senior Division staff. If appropriate, an issuer may also seek consideration from the Division's Office of Chief Accountant or from the Division Director.
 

AUDIT RESULTS

OVERALL RESULTS

Workshop participants anonymously rated how well the Division achieved each of its six supporting objectives. The rating scale used by the participants ranged from 7 (full implementation) to 1 (not being implemented in a meaningful manner). Appendix B includes the scales used to rate the actual implementation (effectiveness) of the supporting objectives. The composite ratings for how well the participants felt that the Division actually achieved its six supporting objectives were as follows:

Communicate Staff Findings - 5.2

Management Review - 4.8

Communicate Comments - 5.7

Resolve Comments - 5.3

Program Performance Measures - 3.6

Staff Performance Measures - 4.1

An important theme, expressed overwhelmingly by the participants, was the personal commitment of Division employees to maintain high standards of quality in filing reviews. Most of the participants also indicated that they felt a strong sense of responsibility to the investing public and a desire to live up to the Division's, as well as the Commission's, expectations. These factors are crucial to an effective program.

The composite ratings above of actual achievement by the participating managers and staff indicated that, except for program performance measures, the supporting objectives were generally being achieved, although some obstacles were impairing full implementation. Both management and staff gave low ratings to the achievement of the program performance measures objective (management - 3.90, staff - 3.32). Workshop discussions indicated that this supporting objective received low scores from participants because of concern with the Division's use of number of filings reviewed as its primary measure of program performance.

Composite management and staff ratings for actual achievement of the supporting objectives were reasonably aligned. Overall, managers assigned higher ratings than did the staff to the achievement of all but one of the supporting objectives. Despite the differences in the ratings, the participants believed that, taken as a whole, the Division was achieving its primary objective to improve disclosure to investors.

Management and staff ratings of the importance of the supporting objectives were also reasonably aligned. Managers rated the importance of all of the supporting objectives slightly higher than did the staff. Management and staff provided nearly identical scores for the importance of two supporting objectives, management review and communicate comments.

For at least the past year, the Division received feedback indirectly from companies on their filing review experience. The Division periodically provided the Chairman of the SEC a list of companies with recently effective registrations. In turn, the Chairman contacted some of the companies for feedback on their experience with the Division's filing review process. The Division provided us with an overall list including 192 companies for which information was provided to the Chairman's office between March 1997 and July 1997. Only three of the companies on the list raised issues concerning the Division's filing review process that resulted in a written response from the Division to the Chairman's office. In all instances, the Division confirmed that the appropriate actions had been taken during the filing reviews.

After reviewing this information, the Office contacted an additional five companies, not on the Division's list, for their feedback on the Division's comment letter process. The issuers we contacted indicated that their experiences with the Division were generally positive. Comments were received timely and the issuers were able to resolve their comments in time to execute their transactions. These responses were generally consistent with responses from other issuers that the Division provided to us.

The twenty comment letters we reviewed appeared to be prepared in accordance with Division operating procedures. We also reviewed reports of 1933 Act registrations filed between October 1, 1996 and June 31, 1997 (1,228 registrations receiving full reviews and 567 registrations receiving a monitor for one or more specific items) to determine the timeliness of the initial filing review comments. Our review results indicated that the Division issued comments within its 30 day timeframe on approximately 88% of the registrations in our sample receiving full reviews and approximately 98% of the registrations receiving a monitor for one or more specific items.

COMMUNICATE STAFF FINDINGS

Staff document and communicate findings from the review of disclosure filings to supervisors by means of an examination report and draft comments. Ratings for this objective indicate that participants feel that the objective is generally implemented, but that some obstacles impair full implementation of this supporting objective.

After reviewing a filing, staff prepare an examination report summarizing the transaction and identifying any material disclosure issues. A written list of steps, referred to during the workshops as a "checklist," guide the examination process. The checklist is a response to a prior General Accounting Office recommendation that the Division document its review of filings.

The checklist is generally the same for all the industry groups. However, in certain industries, such as real estate, banking, and insurance, the checklist is augmented by specialized requirements for that industry called industry guides. Other industry groups have manuals containing guidance on issues specific to that industry. Some industries have no specialized guidance. Participants emphasized that these checklists and guides, while useful, are not a substitute for the ability to understand the transaction in the filing. Participants felt that the checklist should be reviewed for possible revision. Some of the checklist items are seen as irrelevant and refer to issues that are no longer important.

Participants were asked to describe risks in the examination report/draft comment process. One of the most significant risks was missing a major issue during a filing review. However, review of filings and comments by reviewers mitigate this risk. Another risk identified was associated with supplemental materials requested from issuers in response to comments. This supplemental material is sometimes proprietary information that should be protected from public disclosure. However, issuers often do not request this protection. Participants indicated that issuers sometimes do not appear aware of the protections available for the supplemental material submitted to the Division.

The Division has training for new staff examiners and accountants. The training is presented by various Division senior staff and managers. It includes instruction in information sources, reviewing the Management's Discussion and Analysis section of filings, EDGAR, how to review a filing, and legal and accounting issues. Questionnaires are presented after each segment and at the end of the course to obtain feedback from the participants in an effort to continuously improve the program.

Participants were generally pleased with the training they received. They generally rated training on new releases as good. They also felt that the training manual was improved. In general, the mentoring program, where senior staff accountants and examiners provide guidance to more inexperienced staff, was well received, although the quality of the individual mentoring experience depended on the quality of the mentor. However, they also indicated that the training needed to focus more on the work that the staff actually performed. For instance, the training concerned with accounting proposals, while interesting, was not viewed as particularly useful. Also, broader access to information on issues previously identified in a particular industry would also be helpful. Participants indicated that a more direct link between the training and the filings they actually review would make the training more effective.

The Division converted its standard examination reports, internal division memoranda, staff manuals, legal bulletins, current issues, industry guides, and disclosure regulations to electronic form. The electronic documents were then placed on the Division's file server. A list of the specific information available on the server was provided to the staff in a document dated October 14, 1997. Division staff are now able to access the applicable guidance on-line. Also, updates will be provided on-line.

Recommendation A

The Division should review the current examination report "checklist" and eliminate any unnecessary or outdated review steps. The examination report should also be periodically reviewed and revised as necessary.

Recommendation B

The Division should remind issuers of the rules available to protect any proprietary supplemental information they provide to the Division and the procedures used to request the special handling.

MANAGEMENT REVIEW

Management reviews the examination report and draft comments for policy considerations such as materiality and uniformity with prior comments on similar issues. Generally, this task is delegated to senior accountants and examiners. Participants indicated that these senior staff, referred to as "reviewers," are appointed based on their experience and background in the particular group. Assistant Directors (ADs) review the most complex filing issues.

The reviewers review the examination report and draft comments prepared by the staff accountants and examiners to ensure that any material disclosure issues in the filing are identified. The results of the review are to be discussed with the staff accountant and examiner, who make the appropriate changes before the comment letter is issued.

Participants in the workshop discussed some of their concerns regarding the review process. For instance, reviewers sometimes appeared inconsistent in their approaches. Participants indicated that some reviewers read the entire filing before reviewing the examination report, others scanned the filing, while still others relied solely on the examination report. Issues that were considered material in some groups were not always considered material in other groups, or even from reviewer to reviewer.¹ No written guidance has been developed for reviewers.

Also, some reviewers are seen as not willing to discuss their review results with the staff. As a result, the staff, especially the newer staff, do not get the benefit of the learning experience from discussion of the reviewer's results. Some participants felt that some reviewers change the comments on a filing, or insert new comments, without notifying them. These participants described instances where they were embarrassed when an issuer contacted them about a comment they knew nothing about, because the reviewer inserted it without their knowledge. Management indicated that the reviewer should give the review staff a copy of the comments that were issued. The staff should then review what was issued and obtain any necessary explanations.

Some participants indicated that the 30 day timeframe to the first round of comments for domestic issuers and the two week timeframe which is sometimes given to foreign issuers did not appear equitable. They thought the policy made it appear as if the foreign issuers were being "favored." Management indicated that review of foreign filers was facilitated in an effort to encourage foreign issuers to register their securities with U.S. markets. Currently, the two-week timeframe is only provided in unusual cases. Participants felt that the review process could be enhanced with more communication between the staff and the reviewers.

Recommendation C

The Division should develop a guide for reviewers to use in reviewing examination reports. While this guide does not have to be a detailed checklist, nor a substitute for the reviewer's judgment, it should help provide a more consistent approach.

Recommendation D

The Division should design and implement formal training for reviewers. Senior and middle managers and support offices such as Chief Counsel and Chief Accountant should participate in training. This training should result in a more consistent approach to review as well as improved communication.

COMMUNICATE COMMENTS

Participants rated communication of comments as the most important of the six supporting objectives. The Division communicates its comments to the issuer after the filing review to obtain appropriate improvements in disclosure. Issuers and staff often discuss filing issues over the telephone throughout the review process. The review staff faxes comments to the issuer, generally with a follow-up copy by mail. Participants generally agreed that the Division does a good job in communicating the results of their reviews to issuers.

Staff participants expressed the desire for Internet e-mail access. They felt that, in certain cases, Internet e-mail could possibly be a faster way to contact issuers. The Division has already gotten approval for Internet e-mail access. The applicable policy is dated August 19, 1997.

Several participants suggested separating legal and accounting comments in the comment letter. They felt that segregating the accounting and legal comments in the comment letter would make it easier for the issuer to contact the originator of the comments. In response, management indicated that accounting and legal comments were, at one time, presented separately in the comment letters. However, some duplication of comments occurred in categories that included both accounting and legal issues, such as in the Management's Discussion and Analysis (MD&A) section of the filing. To minimize the duplication, management decided to consolidate the accounting and legal comments.

RESOLVE COMMENTS

Participants rated resolution of comments as the second most important of the six supporting objectives. Workshop participants agreed that the current process works well in resolving comments. Issuers who disagree with the views of the Division staff may contact successively higher levels in the Division, including the Director. Although issuers do not always agree with the Division's proposed changes, they generally still make them. Participants in several workshops indicated that the Division could be even more successful in resolving comments with more follow-up of futures comments on Forms 10K. Sometimes the review staff changes or the workload becomes such that no time is available for follow up on futures comments.

Participants felt that the EDGAR (Electronic Data Gathering, Analysis, and Retrieval) system should be improved to make it more useful in filing reviews. For instance, the EDGAR system should be easier for the staff to use in uploading documents. Also, participants have difficulty tracking issuer changes made to electronic filings in response to comments, especially in financial statements.

The Commission is currently seeking a contractor to modernize EDGAR. Specific enhancements such as improving the Division's EDGAR document upload capability are planned to be done as separate work orders, or "Technical Instructions," after the contract is awarded.

Recommendation E

The Division should review the role of futures comments to determine if there is any adverse impact on disclosure if there is no follow-up, and take appropriate action.

Recommendation F

The Division should ensure that the new EDGAR system includes improvements in document uploading capabilities and in identifying and tracking issuer's changes to filings, including in the financial statements.

PROGRAM PERFORMANCE MEASURES

Program performance measures are used to measure and evaluate the performance of the review of filings. Participants agreed that the Division uses the number of filings reviewed as its primary performance measure. Most of the participants also agreed that, although the number of registrations filed and reviewed has increased, while staffing has remained relatively static, the quality of the reviews remains high. Participants also indicated that the result of their activities can be expressed better than relying solely on the number of filings reviewed.² The Division has indicated that its reviews resulted in identification of novel and unique accounting issues, changes in industry practices, and changes in Generally Accepted Accounting Principles.

Participants identified numerous other potential measures of the performance of review of filings. These potential measures include: the cost of the filing review function versus the dollar volume of the markets; the cost of the review as a percent of the filing fee; the dollar amount of changes in financial statements based on filing reviews; the number of material comments on filings with unqualified ("clean") audit opinions; the number of filings with comments implemented; the number of foreign companies that file with the SEC; and the value of fraudulent offerings stopped.

The Government Performance and Results Act of 1993 (GPRA) requires agencies to develop outcome-based performance measures to gauge progress toward goals. An outcome measure assesses the actual results, effects, or impact of a program activity compared to its intended purpose. The number of filings reviewed measures the amount of work performed by the review staff, but does not gauge the effect of the comments on filing disclosures as well as outcome-based measures.

Recommendation G

The Division should use more outcome-based program performance measures to gauge the success of the comment letter process.

STAFF PERFORMANCE MEASURES

Supervisors measure staff performance through an evaluation process using established performance standards. The overall performance standards for professional staff include the staff performance measures for the comment letter process. The standards include six general categories. Each general category contains an average of five sub-elements. Sub-elements relating to the comment letter process are distributed throughout the six general categories. The evaluation process based on these standards is intended to improve individual and staff performance, including performance in the comment letter process area. One of the general categories include a sub-element regarding number of filings reviewed. This sub-element requires a specific average number of "...exams or other reviews per month." The Division also uses the number of filings reviewed to justify its budget staffing levels.

Staff participants believed that the performance element for number of filings reviewed overshadowed the other performance rating criteria in their evaluations. In fact, some participants felt they were penalized for their lower production numbers resulting from processing larger, more complex filings than those used in setting the performance standards. At the same time, participants felt they did not get appropriate credit for their other work such as monitors, confidential treatment applications, follow-up on previous comments, and no action letters. Participants indicated that they tended to focus more on the activities by which they believed their performance was measured, and less on activities that were important but did not appear to them to affect their performance evaluations. They also indicated that the quality of the reviews sometimes suffered in the push to achieve numerical goals.

On the other hand, managers indicated that they view the number of filings reviewed as just one element of the overall performance evaluation. However, adequate review coverage of issuer filings, given limited staff resources, is essential to the mission of the Division. According to management, the benefit of improving reviews of a smaller number of filings would not justify the loss of reviewing a larger number of filings, albeit at only the current quality level. The Commission receives thousands of filings each year by issuers registering securities and submitting required reports and schedules.

Participants also gave us their ideas of the characteristics of outstanding staff. Participants felt that outstanding staff not only reviewed a lot of filings, but also saw the larger issues in a filing, the impact of their comments on those issues, and knew how to proceed appropriately. They demonstrated a high level of interest in the issues, knew when to push an issue and when not to, were articulate, and knew the technical literature. Participants indicated that improving these characteristics in the staff as a whole would also improve the comment letter process.

We asked participants for their opinions on the pluses and minuses of working for the Commission, specifically the Division. Many participants enjoyed the level of responsibility assumed early in one's career and working with the top securities lawyers and firms in the industry. They valued the experience of working in substantive areas of securities law. Participants also valued the extent to which they exercised judgment in their work. In addition, they described a "collegial" atmosphere. All of the participants felt that their work made a difference and had an impact on the securities industry.

On the other hand, staff described a number of concerns. Lack of flexibility in work hours concerned some staff. Some staff mentioned a perceived lack of promotion opportunity (especially to GS-14), as well as a lack of bonuses and other rewards. However, despite these concerns, participants indicated they were proud to work for the Commission, in particular the Division.

Recommendation H

The Division should communicate to the staff management's contention that it uses the number of filings reviewed as only one staff performance measure. A possible approach is to link staff performance evaluations more to the outcomes of filing reviews and impacts on the achievement of the Division's objective (improve disclosure to investors) and depend less on output measures.

[APPENDIX A]

CONTROL SELF ASSESSMENT METHODOLOGY

The Control Self Assessment (CSA) methodology used consists of four primary tasks: identify management objectives, convene workshops to discuss and rate each objective, evaluate the workshop data, and prepare an audit report.

MANAGEMENT OBJECTIVES

At the beginning of the audit, the Office audit staff worked closely with management to develop concise objectives for the comment letter process. Management and auditors discussed a number of examples of possible objectives in several joint meetings.

The objectives selected for evaluation were those that management thought were both important and for which evaluation data would be useful. Management revised their objectives further based on experience gained in the pilot workshop, which consisted of senior Division management.

The primary objective developed by management was:

Improve Disclosure to Investors

Management also developed six supporting objectives, reflecting the activities that make achievement of the primary objective more likely. They were:

Communicate Staff Findings - Document and communicate staff findings from the review of disclosure filings to supervisors by preparing examination reports and draft comments.

Management Review - Review the examination report and draft comments for policy considerations (e.g., materiality, uniformity).

Communicate Comments - Develop and communicate (written and verbal) comments to the issuers or their representatives.

Resolve Comments - Gain issuer acceptance of comments through implementing amendments and futures comments.

Program Performance Measures - Measure and evaluate the performance of the comment letter process accurately.

Staff Performance Measures - Evaluate staff performance in the comment letter process to improve individual and program performance.

THE WORKSHOPS

Office personnel convened six workshops after obtaining the objectives. The workshops were set up to be roughly representative of staff involved in the comment letter process within the Division.

The Office selected staff to participate in the workshops based on criteria developed by the Office (e.g., must have been on staff at least six months). We designed the workshops to be homogeneous with respect to management or professional staff to facilitate subsequent comparisons.

Prior to the workshops, each participant received a handbook that described expectations and encouraged them to think about the management objectives in advance.

Also prior to the workshops, the auditors developed sets of questions that related to each objective. The questions gave structure to the discussions and facilitated the conduct of the workshops.

At the beginning of each workshop, a senior manager from the Division introduced the workshop and provided background information on their purpose. Each workshop took from 4 to 6 hours to complete.

During each workshop, Office staff summarized participant discussions regarding successes, obstacles, and recommendations for each objective. We summarized the comments on flip charts.

After discussing an objective, participants used 4x6 inch cards to anonymously rate, using a scale of 1 to 7, how well the Division achieved the objective. [Appendix B contains the rating criteria used by the participants.] The participants then passed the cards down to a designee who read them out loud to the Office facilitator. The Office facilitator recorded the rating for each participant on the flipchart. The facilitator then repeated the procedure to rate how important (or desirable) the participants thought the objective was. We collected two sets of ratings (i.e., actual and importance) for each of the objectives.

After each workshop, the auditors typed the summary successes, obstacles, and recommendations from the flipcharts to a formatted word-processing document (one page per objective). We asked participants to review the worksheets relating to their workshop and verify their accuracy.

DATA EVALUATION

Quantitative Analysis

The Office keyed the anonymous assessment ratings of the workshop participants into Excel spreadsheets for analysis. We made several comparisons (e.g., how well ratings aligned). The comparisons included analysis of management versus staff, actual achievement versus importance, and opportunities for improvement.

Qualitative Analysis

Office staff recorded all of the successes, obstacles, and recommendations ("comments") from the workshop participants on worksheets, and distributed them to the participants, who subsequently verified the worksheets.

Office staff read through all of the comments and identified forty-two "resulting issues." Each success, obstacle, and recommendation was then coded and electronically transferred to a "resulting issue worksheet." This brought all comments about a particular issue together in one document and facilitated discussion and evaluation of the issues.

Division management and the Office staff reviewed the forty-two resulting issue worksheets. They met twice to discuss the issues and identify potential actions to enhance the comment letter process. This audit report is based on these analyses.

Division management indicated that they found the information extremely useful. In many instances, they had already recognized the issues and taken corrective actions. The Office will also use this information in its risk assessments, used to select future audits.

AUDIT REPORT

The Office prepared the audit report. The "resulting issue" worksheets were given to management to review. Their comments and the results of meetings to discuss the worksheets preceded the preparation of the audit report. This departure from the traditional audit report comment process provided input at an earlier stage in the writing process. This also gave management a better understanding of the audit results, since they also analyzed the resulting issue worksheets.

[APPENDIX B]

EFFECTIVENESS RATING SCALES

ACTUAL EFFECTIVENESS

7         The Commission is successfully implementing the supporting objective. Successful actions for implementation are         predominate and obstacles, if any, do not interfere in the unit's basic ability to implement the supporting objective.

6         Midway between 5 and 7.

5         The Commission is generally implementing the supporting objective. Several successful actions for implementation exist, but some obstacles are impairing the unit's ability to fully implement the supporting objective.

4         Midway between 3 and 5.

3         The Commission is generally not implementing the supporting objective. Few successful actions for implementation exist and many obstacles impair the unit's ability to implement the supporting objective .

2         Midway between 1 and 3.

1         The Commission is not implementing the supporting objective in a meaningful manner. Very limited successful actions for implementation exist and obstacles are so prevalent that the unit is significantly impaired from implementing the supporting objective.

IMPORTANCE

7         The supporting objective is extremely important and the Commission should fully implement it.

6         Midway between 5 and 7.

5         The supporting objective is relatively important and it should be generally implemented at the Commission.

4         Midway between 3 and 5.

3         The supporting objective is relatively unimportant and its general implementation should not be a high priority at the Commission.

2         Midway between 1 and 3.

1         The supporting objective is not important and should not be implemented at the Commission.

Endnotes

¹ Division is conducting a study simultaneous with this audit in an effort to increase uniformity.

² For example, the Division contends that as a result of staff comments in FY 1996, forty-one companies were required to restate their income. Of these, 90% resulted in changes in income of 10% or more. These outcomes of staff filing reviews were documented in "quantifiable benefits" memos. Also, in a 1996 speech, senior Division management indicated that staff comments resulted in increased issuer attention in a number of disclosure areas, including accounting for disposal of assets as discontinued operations, presentation of overly broad geographic segments, and use of valuation allowances. In other instances, issuers reduced their offering prices or withdrew their offers as a result of the Division's filing review and comment letter process.

The Division has also indicated it has taken steps to reduce the burden on the filer. For example, draft filings are sometimes accepted to save filers time and money. Also, in August 1995, the Chairman organized the Task Force on Disclosure Simplification to review and streamline regulations relating to forms and disclosure requirements. The Task Force's recommendations have thus far resulted in elimination of forty-six rules and six forms.
_______________________