Comments of Professor Sam L. Savage on S7-16-02

From: Sam L. Savage
Consulting Professor of
Management Science & Engineering
Stanford University

July 18^th, 2002

To: U.S. Securities and Exchange Commission
450 Fifth Street, NW
Washington, DC 20549-0609

Re: Release Nos. 33-8098; 34-45907; International Series Release No. 1258
File No. S7-16-02

Dear Sirs:

I applaud your attempt to improve the transparency of companies' financial disclosure, and to better communicate sensitivity in financial reporting. However, the steps proposed in release 33-8098 would only further complicate a seriously flawed system. The problem is that in cases involving uncertainty, accounting estimates are often inherently worthless. I will first explain why this is the case, and then describe how a growing number organizations and investors are using simulation models to account for uncertainty more faithfully. This suggests a road to reform, which could provide benefits to both investors and the firms that must report their financial conditions, as well as reduce the opportunity for misrepresentation.

The Flaw of Averages

An estimate in the context of a financial statement is a single number, often an average or expected value, used to represent the value of an uncertain quantity such as future revenues or expenses. Traditionally, an uncertain quantity has been represented by a probability distribution, a bar graph, representing the relative likelihood of various outcomes. Serious trouble can arise when a single number is substituted for a distribution. I refer to this problem as the Flaw of Averages¹. Although the flaw of averages comes in a variety of forms, the following sobering example is characteristic.

Consider the state of a drunk, wandering back and forth on a busy highway. His average position is the centerline of the highway. Therefore the state of the drunk at his average position is alive, but the average state of the drunk is dead. Illustration: The State of the drunk at his AVERAGE position is ALIVE. But the AVERAGE state of the drunk is DEAD ²

An analogous example in financial statements involves projected revenues and expenses. Consider a firm that expects to receive $500,000 in revenue per month with $400,000 in expenses. Assuming that each month's target for revenue and expense is exactly met, then starting from a zero cash position, net cash would grow linearly as shown. This of course ignores any possible fluctuations in revenue or expense, and corresponds to the assumption that the drunk will stay frozen to the center line. Illustration: Line Chart of Net Cash by Month

If month to month fluctuations in revenue and expense are possible, there will be a significant likelihood of insolvency even if the exact averages were $500,000 and $400,000. Here is an example in which even exact estimates of average revenue and expense do not adequately describe the financial situation of the firm.

A 1993 article on accounting rules for recording uncertain numbers in financial statements exposes a smoking arsenal of additional violations of the flaw of averages that are nonetheless sanctioned by the Financial Accounting Standards Board (FASB).³

A Road to Reform: Monte Carlo Simulation

It is easy to say that we should be using probability distributions in financial accounting, but until the widespread use of desktop computers with spreadsheets it was simply impossible. Today, a technique known as Monte Carlo Simulation has gained acceptance as a unified approach to dealing with uncertainty. Instead of hiding behind a single best "estimate" this technique keeps uncertainty alive by bombarding a spreadsheet model with thousands of random inputs while tracking the resulting outputs. Developed during the Manhattan project during WWII, it is now used extensively in fields as diverse as physics, engineering, health care and finance.

There are two widely adopted Monte Carlo software packages from competing firms that work with Microsoft Excel: @Risk and Crystal Ball®⁴. These programs have now been in use for over a decade in a wide variety of industries for modeling their exposure to various types of risks⁵. More recently Monte Carlo has been adopted by investors themselves to illuminate the future potential behavior of their investments. Financial Engines, co-founded by Nobel laureate William Sharpe⁶ and American Express⁷, are two examples of this latest trend.

Dynamic Financial Statements

Since both firms and investors are becoming increasingly familiar with Monte Carlo simulation, it is appropriate to explore the use of Monte Carlo models as a medium of communication between the two groups in the form of dynamic financial statements.

As an example, the cash flow situation discussed above has been modeled in the attached file, Projections.xls. This contains both a standard cash flow projection based on estimates, and a Monte Carlo version based on random inputs. In the Monte Carlo version, the revenues are assumed to be normally distributed with means of $500,000 and standard deviations of $100,000 while the Expenses have means of $400,000 and standard deviations of $80,000. By pressing the <F9> key, you will see widely differing scenarios like those shown below.

Illustration: Line Graph

Note that in one scenario the enterprise becomes insolvent in March.

Monte Carlo software (which essentially presses the <F9> thousands of times while keeping track of the results) would reveal a 30% chance of insolvency in the upcoming year given these assumptions. Furthermore it would be possible to perform repeated experiments while changing some element of the model. For example, by experimenting with various levels of cash infusion in January, it can be shown that an additional $100,000 would reduce the risk of insolvency by two thirds. However, an investment in excess of $300,000 provides almost no additional risk reduction as shown on the right. Illustration: January Infusion of Thousands

Even if the precise distribution of revenues and expenses were not known, virtually any reasonable distribution would lead to some likelihood of insolvency without a cash infusion. Furthermore, the resulting graph of likelihood versus size of infusion would look qualitatively like the one above. On the other hand, any set of point estimates for which revenue exceeds expense will not even admit to the possibility of insolvency, and therefore the graph does not exist even in the mind's eye. Thus we see again how the common practice of using point estimates in financial statements is qualitatively misleading. Because simulation models are less sensitive to assumptions, that they cannot be as easily manipulated by a few little white lies. To significantly change the qualitative output of a simulation you need to tell a whopper.

Conclusions

The use of single number estimates instead of distributions to represent uncertainty is as erroneous as the Ptolemaic view of the earth as the center of the universe. Release 33-8098, as printed from my internet browser is 72 pages in length. It contains approximately 550 occurrences of the word "estimate" and 48 occurrences of the word "uncertain" or "uncertainty". But it does not contain the word "probability distribution" or its equivalent even once. This is analogous to a 72 page report on the solar system that fails to mention that the planets revolve around the sun. If corporate America were forced to comply with these additional regulations, it would not only waste precious manpower, but would offer even more opportunity for obfuscation.

Instead I recommend the following:

To more accurately reflect the qualitative dynamics of their business, corporations should be encouraged to submit their financial statements as spreadsheets containing formulas that model the most important relationships governing success.
Instead of using estimates, those uncertainties to which the business is sensitive should be expressed as probability distributions, even if only approximate.
Investors should also be encouraged to plug in their own distributions and run their own Monte Carlo simulations when in doubt.

This would benefit corporations, who instead of being forced into the quarterly charade of "earnings management" with all of its flawed estimates, could instead provide realistic ranges of outcomes. It would benefit investors, who would at last be provided a view into the true uncertainties in their investments.

The SEC can play a valuable role in this regard, by establishing standards for 1) the level of detail required in modeling the dynamics of a business, and 2) the degree of accuracy appropriate for estimating distributions of uncertain factors.

I recognize that what I have proposed is a huge step, but I believe the accurate reflection of uncertainty in financial statements would provide a huge benefit. The move to dynamic financial statements using spreadsheet models and simulation will not happen overnight, and ultimately the market must decide whether this approach truly improves investor confidence in publicly traded firms. But given that the firms and investors are already independently moving in this direction, how better could the SEC serve to improve 93the transparency of companies' financial disclosure94 than by investigating this approach itself?

Sincerely,

Sam L. Savage
Consulting Professor of
Management Science & Engineering
Stanford University

DISCLOSURE: The author teaches and consults in the area Management Science which includes risk modeling and simulation. He has also developed analytical software of several kinds, including Monte Carlo simulation.

___________________________________

¹ (see http://www.stanford.edu/~savage/flaw/Article.htm)
² (Illustration from "INSIGHT.xla: Business Analysis Software for Microsoft Excel," 1st edition, by S.L. Savage, copyright 1998. Reproduced with permission of Brooks/Cole, an imprint of the Wadsworth Group, a division of Thomson Learning.)
³ "Expected Values in Financial Reporting" by Johnson, Robbins, Swieringa, and Weil, Accounting Horizons, Vol.7 No. 4, Dec. 1993. A further discussion of this article will appear in 93Accounting for Uncertainty94 by Sam Savage and Marc Van Allen in the forthcoming Fall 2002 Journal of Portfolio Management.
⁴ @RISK is from Palisade Corp. (www.Palisade.com). Crystal BallAE is from Decisioneering Inc. (www.Decisioneering.com). The author has recently developed a simpler Monte Carlo package for teaching purposes and smaller applications: XLSimAE from AnalyCorp Inc. (http://www.AnalyCorp.com)
⁵ See www.Palisade.com and www.Decisioneering.com for a number of case studies.
⁶ http://www.financialengines.com/
⁷ http://home3.americanexpress.com/corp/latestnews/mcarlo.asp

Consider the state of a drunk, wandering back and forth on a busy highway. His average position is the centerline of the highway. Therefore the state of the drunk at his average position is alive, but the average state of the drunk is dead.	²

An analogous example in financial statements involves projected revenues and expenses. Consider a firm that expects to receive $500,000 in revenue per month with $400,000 in expenses. Assuming that each month's target for revenue and expense is exactly met, then starting from a zero cash position, net cash would grow linearly as shown. This of course ignores any possible fluctuations in revenue or expense, and corresponds to the assumption that the drunk will stay frozen to the center line.

¹	(see http://www.stanford.edu/~savage/flaw/Article.htm)
²	(Illustration from "INSIGHT.xla: Business Analysis Software for Microsoft Excel," 1st edition, by S.L. Savage, copyright 1998. Reproduced with permission of Brooks/Cole, an imprint of the Wadsworth Group, a division of Thomson Learning.)
³	"Expected Values in Financial Reporting" by Johnson, Robbins, Swieringa, and Weil, Accounting Horizons, Vol.7 No. 4, Dec. 1993. A further discussion of this article will appear in 93Accounting for Uncertainty94 by Sam Savage and Marc Van Allen in the forthcoming Fall 2002 Journal of Portfolio Management.
⁴	@RISK is from Palisade Corp. (www.Palisade.com). Crystal BallAE is from Decisioneering Inc. (www.Decisioneering.com). The author has recently developed a simpler Monte Carlo package for teaching purposes and smaller applications: XLSimAE from AnalyCorp Inc. (http://www.AnalyCorp.com)
⁵	See www.Palisade.com and www.Decisioneering.com for a number of case studies.
⁶	http://www.financialengines.com/
⁷	http://home3.americanexpress.com/corp/latestnews/mcarlo.asp