Preamble.

This is the third and last in a series of essays about models and competitive strategy.

From All About Models, the first in the series:

• Models describe how we believe the world works.
• Models may be in our heads (mental models) or computer-based.
• Computer-based models are always based on mental models, though they can calculate more and better.
• A model is always involved, though perhaps unconsciously, when we say or compute if we do this we will get that.
• No model is perfect. The test for whether a model is useful is whether we can make a better decision with it than without it.

From What The Model Says, the second in the series:

• What the model “says” is at least as much about its design as about the numbers that come out. For example, a model that knows about costs will give you cost-based advice, nothing more.
• The most important decision we make about our model is to choose how our model should think.
• If your model omits key variables, it will leave you vulnerable to competitors who make use of those variables.
• A key test for whether a model is valid is whether it makes conceptual sense for the problem you want to solve. Many common modeling techniques do not.

Excessively cautious notice. This essay may occasionally sound commercial because in it I describe techniques ACS uses, including one technique proprietary to ACS. The two business war-gaming techniques I’ll describe are practiced by ACS and other companies; each company has its own way of doing things. The third, strategy decision tests, is proprietary to ACS. Related techniques (e.g., Monte Carlo simulation) are not.

End of preamble.

People whisper to horses, dogs, and (according to TV) ghosts. Why not models? The model whisperer — perhaps you, savvy strategist — gently, wisely guides models into shape and helps them achieve fulfillment as oracles of your business’ future.

In our last episode I promised that we’d explore when to use mental and computer-based models in competitive strategy. I further promised that we’d talk about them in the context of three strategy-development techniques: qualitative business war games, quantitative business war games, and strategy decision tests.

In qualitative business war games we experiment with ideas and paradigms embedded in our mental models. In quantitative business war games we use computer-based models to stress-test specific strategy options. In strategy decision tests we run massive what-if simulations to gauge the risks and rewards of going down different paths.

The hypothetical examples I’ll present are actually not hypothetical. Each is an amalgam of multiple war games I’ve conducted in multiple industries, with identifying marks removed to protect confidentiality.

Qualitative business war games

Imagine. You run a business preparing a major change. You’re contemplating something unprecedented, even controversial. Your move will be highly visible and very expensive to retract. If it works, the rewards are great. But there are many interested parties out there, including your competitors, and it’s far from clear which side they’ll take. Their opposition could hurt you badly. Should you make the move?

You’ve thought it through and, although you favor making the move (your mental model at work), you’ve quickly bumped up against the chess-like complexity of your situation. You feel how easy it is to slip into rosy predictions. Your management team is split, some offering passionate support, some passionate opposition (their mental models at work). You and they have debated for some time without reaching a conclusion. Your window of opportunity is closing.

What would you do? Why, you’d run a qualitative business war game, of course.

Business war games involve role-playing. Teams of strategists role-play their business, competitors’ businesses, government regulators, consumers, investors, and so on, as needed for the problem the company wants to solve. Through a series of structured exercises and interactions, the teams construct, debate, select, execute, and react to strategy moves.

In the safe environment of a qualitative war game your strategists get to experience what could happen as they roll out a strategy… or, as a role-played competitor, as they fight against it. A qualitative war game is not unlike a mock trial or a field exercise: it doesn’t ascertain what will happen, but it shows what could happen, with far greater richness than a conference-room discussion. It reveals gaps in your thinking. It surfaces unconscious and unfounded assumptions that could haunt you later.

A qualitative business war game plays out your mental model of what you believe or hope will happen. In addition, it will probably uncover new ideas, favorable and unfavorable. You may choose to explore those new ideas with the quantitative methods I’ll describe later, or by running through another qualitative session. It’s because of those new ideas that I strongly advise you to run at least two “rounds” in a business war game; the second round turns the clock back to the kick-off point of the war game, but with the first round behind them the teams know more than they did before.

By definition, qualitative business war games don’t require computer-based models. They are flexible, time- and cost-efficient, and they can simulate just about anything. They are a great way to generate ideas because the adrenaline rush (I’m not kidding) of simulated competition stimulates creativity. Oh, and they’re fun.

Their drawbacks mirror those of mental models: humans make lousy calculators, even more so under uncertainty, and so it is difficult to get reasonable performance projections in a qualitative war game. Frankly, I don’t even try. I wouldn’t believe the projections.

Quantitative business war games

Imagine. You run a business facing a new competitive threat that’s sure to materialize. You and your colleagues have developed several mutually exclusive countermoves. The countermoves share no common ground and there’s no split-the-difference compromise to be found. Your colleagues advance persuasive arguments in favor of each possible move, and promise future glory. But how can that be, when the moves are so different? How can you know what to expect? What if competitors counter your countermove?

You ponder and reject the usual tools. Financial spreadsheet: doesn’t take competitors into account. (Notice the mental model in those spreadsheets, and refer to our discussion of Model V and Model M in What The Model Says.) Forecast: history won’t be a good guide because the new competitive threat changes too much. (Notice the mental model there too.) SWOT analysis: just a way to express your mental models using four categories.

So you turn to a quantitative business war game. You use a qualitative game when you’re looking for ideas and surprises; you use a quantitative game when you’re looking for analysis and choices. A quantitative war game uses teams and role-playing like the qualitative variety, but it adds (as you might expect) a computer-based model. The model, calibrated for your competitive environment, takes as input the strategy decisions made by the teams and provides as output estimates of sales, market share, profits, and so on.

Sidebar. Tempting and fascinating as it is, I’m not going to discuss the process of building such models in this essay. Contact me at info@whatifyourstrategy.com if you’d like to talk about it. I’ll just say 1) it’s possible to build such models, 2) it’s not as hard as it might sound, and 3) such models easily satisfy the usefulness criterion, to wit, whether you can make a better decision with the model than without it. See, for example, the Shell case at the start of Putting the Lesson before the Test from Wharton on Dynamic Competitive Strategy (Day and Reibstein, editors, 1997). That model projected a nine-figure loss if Shell were to execute a certain strategy. End of sidebar.

In qualitative war games participants are surprised by the actions teams take. In quantitative war games participants are surprised by the outcomes of those actions. Surprises are good, of course, if they’re in the safe environment of a war game.

Here’s a tip-of-the-iceberg illustration that shows the dynamic of a quantitative war game. Teams develop strategies. They all expect to perform well. Unless it begins with a near-monopoly, no team expects to lose market share. Yet each and every market in the world always, always contains exactly 100% market share, which means that for one business to gain share someone must lose share. Without a model, the teams argue endlessly about who gains and who loses. With a model, teams see who gains and who loses, and why. It makes the discussion much more fruitful.

By definition, quantitative business war games require computer-based models. They generate ideas, as do qualitative games, but their key benefit is that they let you evaluate and contrast your strategy options rigorously and objectively. They can produce consensus and action quickly because participants see the results of option A versus option B. (I’ve seen Fortune 500 companies change course overnight.) They help set reasonable targets and expectations. Oh, and they’re fun.

Their drawbacks mirror those of computer-based models: more time and cost to set up, and the model won’t be able to simulate actions not anticipated in the design phase. More subtly, some models are simply better-designed than others, so you must choose carefully.

Strategy decision tests

Imagine. You direct pricing for a business in a highly competitive market. You and your competitors watch each other’s prices like genetically modified hawks. You and they can change prices frequently. How should you set your prices?

This situation is rife with mental models and computer-based models. Some pricing strategists — your competitors? — look at prices periodically and then make a decision what to do. Those decisions, of course, come from some invisible and visceral combination of their mental models and any analysis they see. Other pricing strategists — your competitors? — may use computer-based models that figure out “optimal” pricing based on, for example, demand curves or the status of supply and demand at this microsecond.

So how should you set your prices? This much is clear: if you go too low you may trigger a price war or leave money on the table, and if you go too high you may lose share and long-time customers. If you try to duplicate others’ apparently successful moves you may make matters worse, sort of like grabbing someone else in midair instead of opening your own parachute. And of course prices are relative, and of course your prices affect their prices, and vice versa. So how should you set your prices?

You set up a strategy decision test.

Sidebar. Or perhaps you use other tools, of which there are at least several. As my focus in this essay is models and decision-making, not a pricing-technique critique, I won’t delve into methods other than strategy decision tests. I’ll just reiterate that all computer-based models are based on mental models, and it behooves the model consumer to ensure that a model is conceptually sensible and appropriate for the intended use. End of sidebar.

A strategy decision test is a massive what-if simulation. It’s useful when there are many options for you and/or your competitors and you want to understand the risks and rewards of your options. Each of your options may face many thousands of possible outcomes, and evaluating all your options may mean looking at many millions of futures. (See also The How-Likely Case.) That’s too many to do in your head. It’s also too many to do in a spreadsheet. But it’s not too many for a strategy decision test.

Strategy decision tests look at all the possibilities and summarize the results. They don’t involve teams or role-playing. They do, however, thrive on ideas for strategy options, specifically including ideas generated in war games or by brainstorming.

Obviously, strategy decision tests use computer-based models, and all my usual imprecations about model quality and sensibility apply.

One of the benefits of strategy decision tests is its comprehensiveness, which lets it show a clear view of risk; that is, of the range of what could happen. That comprehensiveness would have been very difficult not many years ago, and simply impossible not many years before that.

Another, quite intriguing, is what Malcolm Gladwell called “serendipity” in his article about drug companies screening millions of compounds to see if they might have useful medicinal effects. (“The Treatment,” The New Yorker, May 17, 2010.) “[Screening] provided a chance of stumbling across something by accident — something so novel and unexpected that no scientist would have dreamed it up. It provided for serendipity, and the history of drug discovery is full of stories of serendipity.” He cites penicillin and Viagra. “What he [a cancer researcher] found was exactly what he’d hoped for when he started his hunt: something he could never have imagined on his own.”

That serendipity happens in strategy decision tests. It’s not only that I’ve seen strategy decision tests find strategies that beat the best I could develop. I’ve also seen strategies pop out that I could never have imagined on my own.

The drawbacks of strategy decision tests are two. First, developing a strategy decision test takes somewhat longer than developing other kinds of computer-based models. Second, the analysis may appear like a black box, not because anything is hidden but because the scope is too big to be eyeballed. Oh, and maybe a third. The results may be fascinating, insightful, and even jaw-dropping — I’ve felt mine fall a few times — but it’s not as entertaining as a business war game.

The n rules for model whispering

My goodness, we made it! Thank you, friendly reader, for staying with me to the end (almost) of this series of long essays.

We end with a few rules to remember as you whisper to your models and make your competitive-strategy decisions.

There is always a model. Computer-based or mental; Model V or Model M or Model XYZ; big or small. You never decide whether to use a model. You always decide which model to use. Don’t expect a financial spreadsheet to give you advice about marketing strategy. Corollary: choose a model on the basis of sensibility for the decision you need to make.

Computer-based models are people too, sort of. Computers only know what people have fed them and they only think what people have taught them. It’s not human versus computer. It’s, again, which model you want to use, and is that model better applied in your head or in a computer. Corollary: don’t try to do too much arithmetic in your head.

Models can stimulate creativity. People get results from models — their mental models, others’ mental models, computer-based models — and they create ideas for how to do better. (War games are a prime example.) They even get ideas by designing models, as they become attuned to the wide range of variables under their control. Corollary: you can build your creativity by asking yourself “how would I model that?” when you come across an interesting strategy. That activity thrills me and makes me endlessly fascinating at parties. I know that because people can hardly wait to wander off and think about it on their own.

Surprise is good. If you can always predict what a model will say, the model doesn’t add any value. The model adds value when it tells you something you didn’t already know. Corollary: if you (i.e., your mental model) disagree with another model, take the opportunity to learn whether to update your mental model or to jettison the other model.

There is no such thing as precision or accuracy about the future. There are many, many, many possible futures. No practical model can capture every relevant variable and tell you which one will happen. (Put another way: a perfect model of the next five years would take five years to run.) Depending on the technology we employ, you can get a good / better / best view of those futures, but no one can guarantee a correct view. Incidentally, there is also no such thing as data about the future, but that’s only one reason why there’s no such thing as precision or accuracy about the future. Corollary: the objective is to improve the odds of making good strategy decisions.

Finally: ask what-if, and imagine. “Whenever you see a successful business, someone once made a courageous decision.” — Peter Drucker

Further Reading

About business war games in general
Business War Games
Learning Faster Than The Competition
The Seven Deadly Sins of Business War Games

About quantitative business war games and methods
Honey, We Shrunk The Industry
Honey, We Shrunk The Industry Again
Precision In, Garbage Out

About strategy decision tests
Predicting Competitors
Strategy decision tests
When I Was Wrong

Preamble. This essay is the sequel to All About Models. In that essay we covered several points:

• Models describe how we believe the world works.
• Models may be in our heads (mental models) or computer-based.
• Computer-based models are always based on mental models, though they can calculate more and better.
• A model is always involved, though perhaps unconsciously, when we say or compute if we do this we will get that.
• No model is perfect. The test for whether a model is useful is whether we can make a better decision with it than without it.

You don’t have to read that essay before this one, but it wouldn’t hurt. End of preamble.

So you’ve got a model, whether mental or computer-based. Or several models, put forth by several people or several computers. Congratulations! Your brainstorming, calculating, visioning, calibrating, and/or programming has paid off. Now you’re ready to get value and insight.

So, what do you do next? How do you get that value and insight?

Not so fast. It’s not so easy. In fact, you may be shocked a few paragraphs from now.

Before we listen to what a model says, we must figure out whether it speaks well-founded wisdom or well-intentioned folly.

Model V versus Model M

In All About Models I posited two models for a large manufacturing business. Both models acknowledged that our business has high fixed costs and that we must cover our costs to be profitable. They diverged from that point.

• One model said costs per unit go down as we produce and sell more units, and that selling more units also brings in additional revenue. We’ll call that Model V, for volume.
• The other model said it’s easier to cover our costs if our prices are high, and that excellent quality and a strong brand give us pricing power. We’ll call that one Model M, for margin.

Model V and Model M could be implemented as computer-based models or remain as mental models. It doesn’t matter right now.

For Model V, “what the model says” is to increase volume. For Model M, “what the model says” is to increase margin. Notice:

By listening to what the model “says,” we listen to the model’s design
at least as much as we listen to the numbers sifting through
our mental or silicon calculators.

We made the most important decision of all when we built the model: we chose which way, volume or margin, is the correct way to think about (i.e., to model) our large manufacturing business. Any subsequent analysis and action issues from that chosen paradigm. If we’ve chosen Model V, for instance, and if our profits underperform, we will look for other ways to boost volume. Why volume? Because the model we adopted, which codified the way we thought, shows us that volume drives profits.

The power to speak

What we hear from the model depends on which model we’ve given the power to speak. Choosing the wrong model messes up a lot, perhaps even everything. (See also Predictable Competitors and Predicting Competitors.)

In this example, neither Model V nor Model M is right. They’re not right because each misses something important, namely, what’s in the other model!

And that’s not all. For example, neither takes competitors’ actions and reactions into account. Sure, your unit sales may go up if you cut your price. But what if competitors cut their prices too? Okay, perhaps total demand will grow. Will it grow enough to compensate for the lower prices? Maybe yes, maybe no. The point is that we won’t even get to that question if our model ignores competitive dynamics.

The situation isn’t necessarily quite as dire as it sounds. We can take dubious, indulgent comfort in the notion that our competitors may make the same mistakes we do. If a whole industry adopts a way of thinking (and many industries do), no single business suffers a competitive disadvantage due to choosing the wrong model. On the other hand, a business could potentially enjoy the benefits of a different model if it can find one. That’s what those upstarts coming out of left field do. That’s what we call “game changing” action. Changing the game means changing our models and starts by changing our minds.

This discussion of Models V and M is highly and vitally practical. But rather than lengthen a series of long essays by reciting my evidence, I’ll just provide a link to real-life war stories. (Look about halfway down that post.) See also the story of Shell that begins Putting the Lesson Before the Test, a chapter from Wharton on Dynamic Competitive Strategy (Day and Reibstein, editors, 1997).

Does it make sense?

There is always a model when we estimate, assert, or calculate that this action will lead to that result.

The challenge in developing an effective strategy begins long before we decide what actions to take and how to execute them. The challenge begins when we decide what model to use. By selecting a model we frame the way we see and evaluate our strategy options.

Prudent strategists want to make sure a model is valid before they select it and thereby entrust their businesses and their careers to its advice. Based on what we’ve discussed about Models V and M, I suggest that “valid” refers first and foremost to whether a model makes sense.

For example:

• A model that looks only at price does not make sense if customers perceive differences in product or service quality.
• A model that calculates changes in market share does not make sense if it ignores customer loyalty, inertia, and switching costs.
• A model that accounts only for our business’ actions does not make sense if we have competitors.
• A model that extrapolates the past into the future does not make sense if we expect (or want) the future to look different from the past.

And so on. Those statements reflect what ACS calls principles of competition.

Three things to notice about that brief “a model that” list. First, none of those statements is particularly controversial. Second, none of those statements is particularly complicated or arcane. Third, according to those statements many of the models commonly used in strategy development violate the does-it-make-sense criterion.

Sensible models are difficult to handle in our heads, partly because of the arithmetic and partly because of the number of we do this, then that happens, then that, then that, then… connections. That’s why I favor computer-based models for many applications. There are times, though, when computer-based models are overkill or even a distraction.

In my next and final essay in this series, The Model Whisperer, we’ll explore when to use mental and computer-based models. We’ll apply those models in qualitative business war games, quantitative business war games, and strategy decision tests (sometimes called decision tournaments).

This essay is full of good stuff about models. Of course that depends on how you define good stuff and models. If you mean something involving delectable (your definition here too) folk lurching down a runway with sour expressions stitched onto their faces, well, sorry, I’m not that kind of manager. But if “good stuff about models” to you means cool and insightful methods to improve decision-making, as it does to me during working hours, then you’ve clicked to the right place.

The fascinating world of models and decision-making

All of us live with models. More accurately, models live within all of us. They help us every day.

As if that weren’t enough, there are models outside us too, and they help us too.

In general, we can classify models chemically: those based on carbon and those based on silicon. The former, mental models, operate inside our heads. The latter, computer-based models, operate inside computers. They behave differently.

Mental models

Mental models are the things in our heads that say if you do this then the result will be that. They reflect knowledge and/or beliefs about how the world works.

Our ability to, say, throw physical objects to a specific target without consciously solving equations comes from mental models. If asked, we could enumerate the key variables involved — the force we apply, the distance it must travel, wind resistance — but we don’t think about them when we throw an object.

Those mental models work well, and without controversy, because thrown objects precisely obey the laws of physics for everyone, every time, so far. But not everything covered by mental models is so consistent. Humans’ mental models can differ dramatically and contentiously where our experience is more varied, complex, or ambiguous. As with competitive strategy.

Imagine you and I run a large manufacturing business. A mental model for what makes it tick may go like this:

• We have heavy fixed costs.
• To be profitable, we must cover those fixed costs and all other costs.
• Fixed costs go down per-unit if we sell more units.
• Selling more units also brings in more revenue.

I’m not saying that mental model is right or wrong. I’m saying only that it is an example of a mental model.

That mental model reflects a certain list of variables (akin to those for throwing physical objects) and it leads its owner to think in a particular way. It focuses on internal operations (our costs), it recognizes the effects of volume (keeping the factory full reduces costs per unit), and it would favor moves such as price cuts or mass marketing because, given the list of variables, those actions would promise to build volume.

I’ve encountered exactly that mental model in the business war games I’ve conducted, and I’m sure you’ve heard it too. It’s common and it at least appears to make sense.

Think again about our large manufacturing business. What would be a different mental model? How about this:

• We have heavy fixed costs.
• To be profitable, we must cover those fixed costs and all other costs.
• It’s easier to cover our costs if our prices are high.
• Excellent quality and a strong brand give us pricing power.

I’m not judging that mental model either. Notice, though, that it reflects a different list of variables and it leads its owner to think in a different particular way. It focuses on customer perceptions (quality and brand), it recognizes the causes of margin (profit comes from the difference between prices and costs), and it would favor moves such as price increases or product differentiation because, given the list of variables,  those actions would promise to build margin.

I’ve encountered that mental model as well, and I’m sure you have too. It’s common and it at least appears to make sense.

Our large manufacturing business is hurting. What should we do? Your mental model says cut price, my mental model says don’t. We battle, we spreadsheet, we PowerPoint. We argue passionately about why a price cut is a good idea or a bad idea, we selectively cite anecdotes and trends and studies. Oddly, we spend less time asking how we can end at different conclusions when we start with the same data; that is, asking how our mental models differ. (See also Predicting Competitors.)

The point is not that any given mental model is right or wrong. (That said, we all know that no one is always right.) The point isn’t even that we must eventually choose one mental model or another. The point is that we all have mental models, we use them all the time, they’re unconscious to us and invisible to others, they appear self-evidently true to their owners, and they differ from person to person.

Mental models have benefits.

• They’re as flexible and creative as the human brain.
• They accommodate new information in real time. We can revise or update them quickly; we just change our minds.
• We always have them with us, they’re always ready to go, and we don’t need help from IT to operate them.
• Mental-model conflicts, annoying as they may be, help us learn. The wisdom of crowds. The marketplace of ideas. Very beneficial in qualitative business war games.

Mental models have drawbacks. For example, they require that we handle numbers in our heads. Sure, we can handle the math behind accurately throwing objects, but try this much simpler arithmetic: what’s 1,248 x 3,579 – 9,876 + 5,555? You’re right, it’s 4,362,271. I was just checking to be sure you knew too. How about this: if our sales are $1,000 today and we expect 3.15% compound annual growth, what will our sales be after 63 months? And we haven’t even gotten into any interesting problems yet. How about our profits after 5 years, assuming that brand popularity rises 9% and fixed costs are… well, you get the point.

It’s not just about handling numbers; it’s also about understanding ripple effects. We cut our price and expect volume to grow, but our competitors want volume too and they feel threatened and so they cut their price in response, so now we’re still at parity but at lower prices, so we cut our price again or cut our costs, and then, and then, and then. Unintended consequences come from unanticipated effects. Scenario-planning and role-playing programs, like business war games, help reveal those effects and consequences. (See also Do Not Overtighten.)

For an amusing and related diversion — and we probably all need one right about now — you might enjoy the lyrics to Tom Lehrer’s classic satire New Math.

Computer-Based Models

Remember that our mental models tell us if you do this then the result will be that. Say you tell about computer about “this” and ask it what “that” will be. Sorry; it will just sit there in stolid silicon silence. Before the computer can tell you about “that,” you have to tell it how. Your computer thinks as you tell it to. Your computer thinks like you.

All computer-based models start in life as mental models.

If you believe you should forecast your sales by extrapolating trend lines into the future, you or your Excelophilic proxy will insert trend-line-extrapolation equations. If you believe you  should forecast sales with a statistical model, you will commission a statistical model. If you ask someone else to figure it out, your computer will think like the person to whom you delegated thinking. And if you believe your mental model is good enough, you won’t even turn on your computer. Let it sit there.

Computer-based models think neither better nor worse than humans. That’s because they think like humans. Their benefits are in clarity, speed, precision, and scope.

• They’re explicit and visible. People can see, discuss, enhance, and ultimately share the way of thinking embodied in the model.
• They compute far, far faster than humans can. That’s more than a convenience. It means also that it’s practical to conduct serious what-if tests with many scenarios. A simulator I wrote calculates about 20,000 scenarios per second, which is handy when I ask it to process millions. (See Millions of Pricing Simulations.)
• They calculate much more accurately than humans can. (See also Precision In, Garbage Out.) Oh, by the way: the arithmetic we covered earlier? The real answer is 4,462,271. The 4,362,271 I mentioned was a sly test. You didn’t know it was wrong? That’s the point.
• They can keep track of many more variables than humans can. A strategy simulator I created for a company could handle roughly a thousand decisions for them and their competitors in multiple countries. Not a job for a mental model. Very beneficial in quantitative business war games and strategy analysis.

Computer-based models also provide a different kind of benefit. Yes, they serve as mental models mated with deft calculators. They also let strategists compare mental models. Back to our large manufacturing business, the one where you and I have been arguing whether to cut price. Use a computer-based model to run it both ways. And maybe some other ways too: cut capacity, broaden the product line, lead prices up, and more.

In other words, computer-based models allow us to run what-if tests. They can do so in quantitative business war games or even in the massive explorations ACS calls decision tournaments. (See also When I Was Wrong.)

Computer-based models sometimes are like rocket science, but their drawbacks are simple. They take time, money, and skill to create. And, of course, they need mental models for their raw material, which makes them vulnerable to lousy mental model in, garbage out. Note that the garbage-out would be the fault of the lousy mental model in, not the computer.

Is the model valid?

I think we’ve established that strategists always use models, whether mental or computer-based. Sooner or later someone asks how you know your model is valid. I think that mostly they want to know about numerical validity and accuracy; does the model work, has it been tested. That’s a good question we’ve all been trained to ask. We’ll talk about that next. However, I think it’s even more important to ask about conceptual validity; in effect, should the model work, does it make sense. We’ll talk about that in the part about rephrasing the question, just a few terrific paragraphs from now.

Actually, people ask the validity question only with computer-based models. Mental models are invisible, don’t feel like models, and are self-evidently correct to their owners. People may contest other people’s conclusions, but, as we said a while back, people rarely ask why they reach different conclusions from the same data.

When people ask if a model is valid they usually mean to ask whether it fits known data. The validity question is fair, but fitting known data is often not a useful test.

It’s always possible to put together a model that fits known data. If you expect the future to look just like the past, then such a model might even be useful. (Assuming it didn’t violate good model-building hygiene, such as using up too many degrees of freedom. See also Predictable Competitors.) But it’s not especially interesting to model a future that will look just like the past. The time you really need a good model is when the future will not look like the past. Unfortunately, there are no data about the future, so the does-it-fit-the-data test isn’t available. (See also More Internet Users than People.)

I recommend rephrasing  the question to shift from numerical validity to conceptual validity. Rather than ask if a model is accurate, ask if it is sensible. Ask how it works, ask what it takes into account, ask on what paradigm it rests. Remember that making strategy decisions is not about accounting, trend lines, or forecasting. It is about strategy, and the models you select should work with strategy concepts. (See also Pundits and Stress and With All This Intelligence, Why Don’t We Have Better Strategies?)

The bottom line

The ultimate point about validating models — and about selecting models and especially about using models — is this. You are going to make decisions no matter what. The relevant question is whether you can make a better decision with the model than without it.

No model is perfect and there are no guarantees of fabulous future performance. Improving the odds of success is all you can hope to get by using models. Fortunately, it is a lot to get. It’s the difference between the gambler and the casino.

This essay talks about models themselves. The next in the series, What The Model Says, discusses which models are worth listening to. The final essay, The Model Whisperer, connects mental and computer-based models to techniques for developing competitive strategy.

This brief essay highlights key themes from my Numbers Gone Wild workshop at the 2010 SCIP International Annual Conference on March 11, 2010. It does not replay the workshop, as the workshop was energetically interactive, verging at times on raucous; well, as raucous as competitive-intelligence professionals get. Not to mention that the group enjoyed numerous punch lines and shockers, and I wouldn’t want to spoil the delights and surprises if someday you experience the workshop yourself. Rather, this essay focuses on concepts and conclusions, which will shock and delight better than would a play-by-play recap.

Introduction: Crazy

Having numbers drives us crazy.

Lacking numbers drives us crazy.

Seeking precision drives us crazy.

Living with imprecision drives us crazy.

In short, we are going crazy.

Numbers surround us. Even people who prefer qualitative methods (“quals”) use numbers every day.

Numbers bug us. Even people who like quantitative methods (“quants”) criticize numbers every day.

In this workshop prudent people explored the unintended ways in which numbers going wild cause us to suffer. We suffer as a result of the bad decisions, and the collateral career damage, that can come from untamed numbers.

How to Like Numbers

I like most numbers. When I say “most” I don’t mean that I get along with all the digits except 7 and 8. I mean that numbers are literally the only way that we can gain certain insights, knowledge, and even inspiration, and that makes them likable.

Of course, the presence of numbers does not guarantee insight, knowledge, or inspiration. On the other hand, the absence of numbers limits us to anecdotes, impressions, and hypotheses. So, we want numbers.

I like most numbers. I don’t like most factoids. Here’s an example: “GM’s market share went down at the same time the reliability of its cars went up.” That statement is true and over many years it was measured precisely in cars sold and defects per hundred vehicles, but it is not likable. Why not? Here’s one of several reasons: because it is context-free. Context would tell us what else was happening while GM improved reliability and lost market share. Context would reveal that many more-reliable Japanese cars had entered the market and the other American producers were improving reliability too. GM almost certainly would have lost more market share if it hadn’t improved reliability.

Our unlikable factoid illustrates how we can decide which numbers to like and which not. The relevant criterion for likability is not whether a number is “actionable.” After all, the factoid is actionable: advise GM management to regain market share by making their cars less reliable. The relevant criterion is whether a number is sensible.

“Sensible” needs a big tent. It includes context, robustness, and quality of analysis. It does not imply or require approval. You might not like a number, but numbers are not popularity contests. (Which is amusingly ironic, if one is easily amused, because popularity contests are judged with numbers.)

The big tent of sensibility also does not imply or require precision. It doesn’t matter a whole lot whether GM’s market share declined 11.2% or 11.3%. No one would breathe a sigh of relief to find the loss was the former and not the latter. Moreover, precision is not even an option for many numbers, such as all numbers about the future.

Notice that the sensibility criterion separates the wildness of a number from what we do with that number (e.g., approve of it or take action on it). That is a good thing. When we humans get wrapped up in strategy debates and decision-making, we confound the goodness of a number (its sensibility) with our approval of the number (its value in getting our idea adopted). We slip into judging the goodness of a number by the support it lends to our idea, rather than judging the goodness of our idea by the support it gains from a number. It is to avoid confounding sensibility and approval that I design the structure of my strategy-simulation models before calibrating the models with data, and that I calibrate the models before simulating strategies with them.

Models in Our Heads!

Think about your business. What would happen to its profits if its prices were to rise 5%?

To answer that question you will execute a series of unconscious cerebral calculations to forecast the net results of many interconnected moving parts. How do I know you will do that? Because you will have an answer to my what-would-happen question, meaning you will promptly predict how prices will prod profits.

Let’s think about that. To promptly predict how prices will prod profits you will assess:

• Competitors’ reactions to your price increase. Will they follow you up or will they tout their new price advantage? How quickly will they move, how far will they go?
• Customers’ reactions to the price increase. What will happen to your business’ sales in the short term? In the long term? Will customers expect a better product for the higher price? Can you market the increase in a soothing way? Will you try to hide the increase via smaller packages or hidden fees? Will customer loyalty dampen or conceal the effects?
• Given the changes (if any) in your business’ volume, what happens to costs? Are fixed costs fully covered? What are the implications for contracts with labor, distributors, suppliers, and shipping? What about sales support and customer service?
• And more.

It would be extraordinarily difficult for you to take all of those factors into account in your head, not to mention doing the arithmetic. And yet you will proffer a prompt prediction to the what-would-happen question. As would most business professionals. You and I hear it all the time; just listen to debates about what works and what to do.

That prediction comes from what’s called a mental model. It’s just like a computer model, except it happens in one’s head and it operates with a lower degree of computational accuracy.

A lower degree of computational accuracy? Yes. Try this. You sell a product for $129 per unit. You sold 51,500 last quarter. The market is growing at a compound annual rate of 3.8%. What will be your revenue next quarter? Write down your answer. That, by the way, is a much simpler question than the one I posed before, the one about raising prices. Even so, you probably had a confident opinion about the first question while feeling helpless to answer the second without electronic assistance.

Mental models are always switched on, even within the quals wandering among us. They matter because we use them to predict the results of actions we might take and to judge the validity of the computer models we might use. We can and should take these steps to use both mental models and computer models without producing garbage-out:

• Mental models are generally unconscious. Make them explicit. Draw them, discuss them, think through the assumptions and principles that drive them. Question mental models and their owners, respectfully, just as we question computer models. Separate their conceptual sensibility from the numbers they produce, and focus on the former first.
• Because humans tell computer models how to think, computer models think like humans. If you believe a financial analysis is appropriate, you use a financial model in which the computer has been taught that profit equals sales minus costs. There’s a flip side: when you use a financial model, you implicitly aver that the problem at hand is properly addressed by financial analysis. Worries about “garbage in” apply to the choice of analysis at least as much as to the numbers fed in.

Important point: both mental models and computer models are about thinking. Both reflect how we believe the world works.

By the way, the answer to the arithmetic-accuracy question is $6,705,733.32.

Oh, Them

Let’s compare models. We’ll go back to the $129 product arithmetic and contrast it with the raise-prices prediction. There’s a critical difference between the two.

• Yes, the $129 product arithmetic is more precise. We have actual numbers. But although it’s a real difference, it’s not a critical difference.
• The $129 product arithmetic can be calculated, monitored, and verified. Another real difference, though not yet critical.
• The $129 product arithmetic implicitly and invisibly implied that the solution was a matter of financial analysis. I reinforced that perception by 1) supplying an answer 2) that was calculated to the penny. The raise-prices prediction implied financial analysis much less, and perhaps not at all. That’s critical.

Review the list of assessments you’d make in the raise-prices prediction. You might assess well and you might assess badly, but the way you would think about that problem would probably differ dramatically from the way you would think about the $129 product arithmetic.

The superficial precision of the $129 product arithmetic nudged you toward a method of analysis. (That’s known as “framing” the problem.) Thus, precision in could cause garbage out. Why garbage? Because the $129 product arithmetic should raise exactly the same issues about competitors and customers as the raise-prices prediction, and using a financial-analysis approach would completely, presumably unintentionally, and potentially disastrously ignore competitors and customers. Oh, them.

Conclusion: Not Crazy

The cure to our initial craziness is not to dispense with numbers wholesale, nor is it to embrace them unconditionally. The way to prevent garbage-out is not to scorn precision, nor is it to demand it. The answer is not either-or or some mythical “sweet spot.”

The answer is to deploy the best of both worlds, thinking strategically and calculating rigorously. Not perfectly or exhaustively, but consciously and vigorously.

That means being careful as we frame problems and choose models. It means remembering that there’s always a model, whether it’s in your head or in your computer. It means taking relevant factors into account, even imprecisely, because omitting relevant factors hurts more than using them imprecisely.

I know it’s possible because I’ve done it in the strategy models I’ve built in my career, and I’m not the only one who’s done it.

Let’s close with lessons from the remarkable Deep Blue.

Garry Kasparov, the world’s reigning chess champion at the time, was defeated in 1997 by IBM’s Deep Blue supercomputer. There are several points we can make about that match.

First, Deep Blue was not programmed with trend lines, gap analysis, or financial fundamentals. Of course not; it was a chess match. So why use trend lines, gap analysis, or financial fundamentals to analyze competitive-strategy problems?

Second, Deep Blue was a machine but it thought like a human. That’s because humans programmed it, with their chess-playing knowledge and expertise. Computers do not contest us; they express us. They are amplifiers for our thinking the way power tools are amplifiers for our muscles.

Third, it’s virtually certain that none of the individuals who programmed Deep Blue could have beaten Kasparov. Their combined talent, though, did. It’s also virtually certain that Deep Blue would have beaten any single person whose knowledge and expertise it contained.

Finally, we know that Deep Blue didn’t win every game in the match, nor (obviously) did Kasparov. But there’s a question deeper than which of them was best. The deeper question is this: who or what could beat a Kasparov/Deep Blue team? The value of numbers and models is not numbers and models. The value is better business decisions that help you succeed.

How to avoid garbage-out on your way to better decisions? Think about it.

Workshop
ACS’ Mark Chussil will deliver a workshop, “Numbers Gone Wild: Or, Precision In, Garbage Out,” at the SCIP 2010 Conference in Washington, DC. See below for an overview of the workshop.

Update: See Numbers Gone Wild (the essay), which shares some themes from the workshop, delivered on March 11, 2010.

Webinars
ACS is planning its upcoming schedule of webinars about business war games, strategic thinking, and other topics of interest to strategists. If you would like to be notified as those webinars become available, please let us know.

The Numbers Gone Wild workshop
I’m just an everyday guy who runs 20,000,000 simulations before breakfast. Doesn’t make me break a sweat, though my computer feels a bit warm. I like numbers and I like learning the things that only numbers can reveal or teach.

That said, numbers also drive me crazy. We’re surrounded by pointless numbers, pseudo-precise numbers, even silly numbers, and as a result we make pointless decisions, pseudo-precise decisions, even silly decisions.

The answer isn’t the quant’s digital paradise. That’s so even though my mini surveys show people believe the best way to get better decisions is to get more-precise data. The answer isn’t the qual’s number-free zone. That’s so even though some of my best friends are innumerate. Rather, the answer is in how we think about numbers and in the numbers we choose to think about.

No statistical expertise is required for this workshop. Quals and quants will both be comfortable and entertained.

This workshop uses a series of interactive exercises and games to demonstrate how our misuse of numbers leads to strategy mistakes. We’ll talk about those mistakes in the context of mental models, precision, spreadsheets, gap analysis, trend lines, paper-folding, groupthink, survivor bias, analyzing novel situations, and the Strategist’s Dilemma. We’ll talk about the mistakes incumbents make that let upstarts win. And no, the mistakes we’ll talk about are probably not those you’re expecting. For example, although garbage in, garbage out is a problem for spreadsheets, it’s almost trivial as these problems go.

This is not a workshop about calculating, decimal points, or the difference between correlation and causation. It is a workshop about thinking, in particular strategic thinking. It is a workshop about getting a fresh view on common challenges. And ultimately it is a workshop about making much better strategy decisions.

See also Precision In, Garbage Out

In this essay we will analyze a difficult problem that’s in the news. It’s about how our beliefs and assumptions guide our analysis of a knotty problem. It’s relevant to anyone who works with data, even if — or perhaps especially if — answers seem obvious.

My home town, New Haven, Connecticut, has been in the news. Not because it’s my home town. Rather, because of a reverse discrimination lawsuit (Ricci v. DeStefano) brought by firefighters against the city. The case is in the news also because Judge Sonia Sotomayor, who’s been nominated for the U.S. Supreme Court, joined her colleagues on the Second Circuit Court of Appeals in a unanimous decision backing the city.

Ricci v. DeStefano is now before the Supreme Court. (Update: for their decision, please see the end of this essay.) I thought it might be nice to bring some cold, rigorous thinking to the hot, emotional case.

The case concerns tests used to promote firefighters to lieutenant or captain. Although the legal arguments focus on which parties have which rights and obligations, the core question (which we might hope is relevant) is whether the tests actually were discriminatory.

Here are the numbers:

• 41 people passed the captain’s exam: 25 white, 8 black, and 8 Hispanic. The city would have to promote the 9 people with the top scores. 7 were white, 2 were Hispanic, none were black.
• 77 people passed the lieutenant’s exam: 43 white, 19 black, and 15 Hispanic. The city would have to promote the 10 people with the top scores. All were white.

The city contends that the apparently too-high percentage of whites being promoted by their test scores is evidence of racial discrimination, and so they threw out the test results. The firefighters say that throwing out the test results reverse-discriminates against those who scored well on a fair test.

I decided to calculate whether the high percentage of white promotions was statistically “too” high. If it is statistically unlikely that whites would do so well and non-whites not so well, we’d have evidence that the tests might have been discriminatory (the city’s position). If the odds are high that it the results could happen by chance or merit, we’d have evidence that the tests were not discriminatory (the firefighters’ position).

Note that analysis can say nothing about the intentions behind the tests. Note also that analysis cannot prove that the tests were discriminatory (or not) in some absolute-truth sense. Still, if the odds strongly favor one side or the other, that should count for something. A reasonable person would and should draw different conclusions about X if the odds of X are 1% and if the odds of X are 99%.
 
I wrote a computer program that looked at every possible way to distribute 41 people in 9 slots (the captain’s exam) and every possible way to distribute 77 people in 10 slots (the lieutenant’s exam). Then, it looked at how many of those possible ways matched the actual racial distribution of the results.
 
There are 350,343,565 possible combinations of 9 winners on the captain’s exam. Of them, 13,459,600, or 3.8%, had 7 whites, 0 blacks, and 2 Hispanics. Another way to look at the results is how many had 7 whites and 2 non-whites. Under that test, 57,684,000 combinations match 7 whites and 2 non-whites, or 16.5%.

In statistical analysis, 5% is a common threshold for “significance;” that is, 1-in-20 odds, a fairly reliable result. (More-stringent analysis uses 1%.) Thus, the 3.8% supports the city’s case, and the 16.5% does not.

Then there’s the lieutenant’s test. There are (this is not an exaggeration or a joke) 1,096,993,404,430 possible combinations of 10 winners out of 77 people. Of that trillion-plus, 1,917,334,783 fit the 10 whites, 0 non-whites outcome. The odds of that are far below 1%; to be exact, the odds are 0.17%. That suggests it was not an accident that 10 whites got the 10 top scores. Why it happened — the test itself, the scoring, self-selection among those who took the test, something else — is a different question, about which neither the analysis nor I make any statement. (I didn’t mention merit as a reason why it happened. We’ll come back to that.) All we can say is that there’s only 1 chance in almost 600 that such an outcome would occur by chance. That’s like guessing a coin toss correctly 9 times in a row: it can happen but you wouldn’t bet on it. Those results support the city’s case strongly.

That said, 1-in-600 odds don’t prove there is discrimination. Those test results could happen by chance, especially if multiple cities use the same test or the same city uses the test multiple times. It’s like winning the lottery if you play enough times or dying on an airplane if you fly enough times. And again, there’s the question of merit.

It gets more complicated. We know that New Haven took pains to create a test that wouldn’t racially discriminate. Assuming that they were sincere and at least partially effective in their efforts, that should raise our confidence that the test results happened by chance (the firefighter’s position), not by discrimination (the city’s). How much should we raise our confidence that the test was not discriminatory? I don’t know. A place to start might be to compare the results of the contested exam with the results of previous tests, or to look at other cities’ test results.

Here’s a different complication: how do we define or discern discrimination? Presumably it would show up as an unfair boost, not unlike steroids, rather than as a blatant gift. Let’s try an experiment. What if, for instance, whites were surreptitiously given slightly higher scores on the tests than blacks or Hispanics? I don’t know how that would be done, but let’s assume that there was a clever way. The average score on the lieutenant’s test for whites was 71.8, for blacks 63.8, for Hispanics 63.6. An unfair boost is one of several possible explanations for that difference. The existence of the difference does not prove the difference was unfair or even statistically reliable, though it begs to be studied more. Regardless, what if we split the difference on the averages and subtract 4 points from every white candidate’s score? How many of the top 10 scorers would be white?

Answer: still 10.

Subtracting 4 points from the whites’ scores is arbitrary, and it could be argued that it introduces clear bias in an attempt to eliminate assumed bias. Even so, 4 points doesn’t change the promotion list. That’s an argument in favor of the firefighters who brought the lawsuit, though it doesn’t prove much.

What if we completely erase the average differences among the groups by subtracting 8 points from the whites’ scores? We’d have 6 white winners and 4 black winners, an argument in favor of the city. And we still haven’t proved much.

Neither the 4-point experiment nor the 8-point experiment proves anything about the presence or absence of discrimination. They merely show the sensitivity of the promotions to presumed systematic bias of a certain number of points. The experiment is about the size of the arbitrary subtraction, not about discrimination. The experiment comes down to whether the experimenter believes that 4 points, or 8 points, or 0 points, or 2.736 points, or 12.345 points, is the right adjustment for the differences in average scores. It’s an analysis based on an assumption. (If there are data that support a real adjustment, that’d be another matter entirely. I don’t know if any such data exist.)

Let’s try another approach. Forty-three whites passed the lieutenant’s exam, along with 19 blacks and 15 Hispanics. Given the much larger number of whites, we’d expect that there would be more variation among them than within either of the other two groups. That happened: there was a wider range of scores among whites. Some of those scores were at the top end. That supports the firefighters’ case.

At last, here’s the merit issue I’ve been promising you. Calculating the 1-in-600 odds started with the implicit and more-or-less invisible assumption that all the people who passed the test got the same score. It’s a direct consequence of treating each of the trillion-plus combinations of winners as equally probable. (Did you spot that assumption? I didn’t until I got pretty deep into my analysis.) In effect, my calculations answered the question “how probable is it that equally qualified people of different races would produce 10 white winners (the lieutenant’s test) or 7 white and 2 Hispanic winners (the captain’s test)?” But how do we know people’s qualifications? That’s what the tests are supposed to reveal. And if the tests reveal merit, then the test results would be right, by definition. But I don’t know if they do (perhaps someone else does); some tests work and some tests don’t. For now, we can only make assumptions.

Presumably the New Haven Fire Department believes the tests measure something of value. On the other hand, presumably no one believes the tests are perfect. So what should we conclude?

Alas, our analysis is not conclusive. If we assume that the candidates were equally qualified, more or less, the 1-in-600 calculation is pretty compelling in favor of the city. Ditto if we assume the tests and scoring were slanted, intentionally or not, toward the white candidates. On the other hand, the wider variation in the larger group, the less-than-overwhelming 1-in-26 odds on the captain’s test, and the city’s previous efforts to ensure fair tests argue in favor of the suing firefighters.

Most important, there’s the question of whether the tests measure merit. If they do, the firefighters’ case is strong. If they don’t, the city’s case is strong.

The bottom line: no definitive answer yet.

Let’s put New Haven aside and up-level the discussion.

When I ran my computer program, I was sure of my conclusion: New Haven is right, the firefighters are wrong. As I wrote this essay, though, I thought about questions my dear readers might ask, and those questions made me think. I questioned my methods, assumptions, and conclusions. I went back and forth as I crunched numbers every way I could imagine short of making this essay my career. So much for cold, rigorous thinking; instead, I got a cold, hard dose of unwanted humility as my conclusion morphed into I don’t know. More data might help, especially information about the validity of the tests. I don’t have those data, and even if I did, there’s only so much time for analysis before we must make decisions. That’s true in government and in business.

That I have not come to a clear conclusion about Ricci v. DeStefano doesn’t mean this exercise (and your faithful reading) has been useless. Quite the contrary; we can come to conclusions about how we come to conclusions. This exercise has:

• Made me less likely to jump to a tempting conclusion based on a short stack of factoids.
• Reminded me that numbers don’t speak for themselves (1-in-600 odds), that analysis reflects basic assumptions (equally probable combinations of winners), and that statistical processes are always at work (wider variations in larger groups).
• Taught me to actively, deliberately look for contrary data and ideas and to keep asking how my analysis could be wrong or incomplete.
• Helped me formulate different ways to solve analytic problems. For example, when there is no clear conclusion, ask a different question. “Which mistake would I rather make” is a good one. “What would a smart reader say” is another.
• Shown me that even though “proof” may be an unattainable standard, we can avoid simplistic answers.
• Proven to me that inconclusive data can lead to more-thorough thinking.

Both sides care about the truth, and we have learned that the data we’ve seen so far are not sufficient to tell us the truth. We have learned that we can debate the promotion patterns and test scores as long as our voices hold out but those data alone do not reveal the truth. We have learned that making a decision relying solely on the data we’ve seen so far will be at least partially the triumph of persuasion or ideology. Finally (and this is both important and exciting), we have learned what else we need — data on the validity of the tests themselves – to make a decision using reason and analysis.

Update. On June 29, 2009, the Supreme Court ruled in favor of the firefighters in a 5-4 decision written by Justice Anthony Kennedy. The Wall Street Journal reported that Justice Kennedy said employers “must show a ’strong basis in evidence’ before ignoring results of employment-related tests.” As we saw in this essay, the evidence was mixed, and whether discrimination had been present or absent, it would have been difficult to prove or disprove.

Further reading
Steven D. Levitt and Stephen J. Dubner, Freakonomics.
Leonard Mlodinow, The Drunkard’s Walk.
John Allen Paulos, Innumeracy.
Jay Russo and Paul Schoemaker, Decision Traps.
Nassim Nicholas Taleb, Fooled by Randomness.
See also Marvelous Techniques.

We ran ACS’ Decision Tournament™ for pricing in conjunction with SCIP Oregon. You can see a description of the tournament along with interesting results and unique charts.

To date, 156 strategists have participated in the pricing tournament, leading to over 5,600,000 simulations (you’re right, it’s not an Excel spreadsheet) and our shocking blog post When I Was Wrong. If you would like to experience the simulation too (and, afterward, get a white paper designed for simulation participants), please write to info@whatifyourstrategy.com. It’s free and fun.

(Update. As of early 2010, nearly 300 strategists have participated in the tournament.)

All 156 participants in the tournament wanted to win and thought their strategies would win. How do we know? If they didn’t think their strategies would win, they would have chosen different strategies. Nonetheless, the outcomes of their strategies varied hugely. Testing their strategies through strategy simulation — in effect, a massive business war game conducted inside a computer — helps identify which strategies do well and which do badly.

In these times of economic crisis, with the stakes so high on pricing and other decisions, there’s more need than ever before to act creatively, and to stress-test action options before risking time, money, and careers on them.

We thank Sean Campbell and Scott Swigart of Cascade Insights, who chaired and organized the SCIP Oregon meeting, for inviting ACS to conduct the simulation.

Praise be to him, or her, who groweth revenue, and expandeth profits, and raiseth up the stock price after a slump, for she or he shall be exalted as the One who delivereth the business model of the future, and he or she shall appear on the evening news, and shall writeth a book, and shall receiveth a share.

Woe be to him, or her, who declineth after a streak, and presideth over a fall, and misseth the targets, for she or he shall be reviled, and ridiculed, and cast out, and subjected to a harsh but blessedly temporary media stoning, and caused to serveth as a symbol to frighten future generations from the ways of the false profits.

And the praise and woe may or mayeth not help your business.

Imagine that you’ve got 10 ladies and gentlemen hard at work, each of them saying yes or no on some decision. Perhaps they’re assessing applications for credit; perhaps they’re connecting job seekers with job givers; perhaps they’re deciding which shipping service to use for shipments; perhaps they’re recommending products for customers. Imagine further that each of them makes 4 such decisions each work day. That adds up to 10,000 decisions per work year. Imagine that they average 60% accuracy: that is, they say yes 60% of the time that they should say yes, and they say no 60% of the time that they should say no. You know that you’ll get 6,000 correct decisions each year, and 4,000 mistakes.

You have two questions. First, a marvelous technique has been offered to you that can raise their accuracy to 80%, and because it’s costly you wonder if it’s worth it. Second, when you test the marvelous technique you notice that Aaron and Abigail are on a streak while Barry and Bonnie are in a slump, and you want to know how (and whether) to respond.

Is a marvelous technique worth it?
We know pretty well how to assess the financial impact of a product or service that purports to reduce costs. You take the cost savings per use, multiply by the number of uses, and compare the result to the cost of the product or service. If the savings exceed the cost of the product or service, it makes economic sense. There’s a little more to it, such as the time value of money and opportunity costs, but the basic idea is savings versus cost.

What about products or services that purport to improve the quality of our decisions?

People ask my colleagues and me about the value of business war games, strategy simulations, crisis simulations, competitive intelligence, market research, win/loss analysis, advertising, and more. Many want to know how confident they can be that one of those marvelous techniques will improve their bottom line, market share, quality of decision-making, crisis preparedness, or whatever they care about.

Confidence is within reach; certainty is not. Even if a marvelous technique helped every time it had been applied in the past, even if ecstatic executives gushed with testimonials to its powers, even if your situation seems a perfect DNA match for its capabilities, that would not prove you will benefit from it. As we learned from Nassim Nicholas Taleb, the abundance of white swans does not prove the non-existence of black swans.

The important (and answerable) question is whether the likely benefit of a marvelous technique will outweigh its costs. “Likely benefit” is how much better (if at all) you will probably perform with a marvelous technique than you will probably perform without it. The costs include relevant outlays for training, software, hardware, retooling, and so on. Conceptually it’s not so different from looking at the financial impact of a cost-reducing product or service.

We may consider four conceptual outcomes to a decision, each of which has financial consequences:

1. You said yes and you should have said yes. In other words, yes is the right answer, and you got it right.
2. You said no and you should have said no. No is the right answer, and you got it right.
3. You said yes and you should have said no. You did something that you should not have done.
4. You said no and you should have said yes. You did not do something that you should have done.

Got-it-right decisions 1 and 2 will generally have better consequences than got-it-wrong decisions 3 and 4. That doesn’t necessarily mean that it’s worthwhile to spend money reducing got-it-wrong decisions. If outcome 3 is infrequent or if it isn’t a whole lot worse than outcome 2, and if it’s expensive to tell the difference between type-2 and type-3 situations, then it may be cheaper overall just to say “yes” all the time. For instance, a full-service store that takes product returns may prefer to make a few mistakes (taking back a product that they didn’t sell or that was shoplifted) rather than annoy legitimate customers with rigorous checking.

Let’s put some numbers on the problem. There’s no magic in these numbers; we’ll use them merely to illustrate the process and analysis.

• The payoff of correctly deciding yes is $15 (you do something and it pays off) and of correctly deciding no is $0 (you correctly do nothing).
• The cost of incorrectly deciding yes is $7 (you do something and it hurts) and of incorrectly deciding no is $2 (you didn’t do something and it hurts).
• The right answer is “yes” 50% of the time.
• The odds of making the right decision with your current process are 60%.
• The odds of making the right decision with the marvelous technique are 80%.
• The one-time cost of the marvelous technique is $500 (for buying a software license or paying for a research project or some such thing), and…
• …it costs $0.50 to use on each decision (due to fees or longer time to run the new system).

With those made-up numbers, what works?

Always yes. If you always decided yes, you would average 50% x $15 + 50% x $-7 over the long term, which is $4.00. It wouldn’t be that exactly for the same reason that flipping a coin n times is extremely unlikely to be precisely 50% heads and 50% tails, but it will be close.

Always no. It matters on which decisions you’re right. If you’re only right on “yes” decisions (which you’d do by choosing yes every time), you’d get that $4.00 on average. If, however, you’re only right on “no” decisions (by choosing no every time), you’d lose money. (That’s from 50% x $0 + 50% x $-2.) So, with the numbers in this illustration, an always-yes decision strategy beats an always-no decision strategy.

The current process. Your team is right 60% of the time with your current process. Over the long term, that means that when yes is right, they’d decide yes 60% of the time and no 40% of the time, and when no is right, they’d decide no 60% of the time and yes 40% of the time. Since yes is right 50% of the time and no is right 50% of the time, your long-term expectations would be:
                50% x (60% x $15 + 40% x $-2) + 50% x (60% x $0 + 40% x $-7)
which is $2.70. Now that you think about it, you’re not so sure that the team is adding value. Saying “yes” every time averages $4.00, and your team, loyal and dedicated as they are, averages $2.70. Hmmm. (This is why you’re looking for a marvelous technique.)

Perfection. If you always decided correctly, you would average 50% x $15 + 50% x $0 over the long term, which is $7.50. Always-yes is the best simple strategy available (given the payoffs and costs above), yet its $4.00 performs way below the optimal strategy’s $7.50. That means it may be valuable indeed to find out how to make better decisions. Will the marvelous technique do the trick? Will it at least beat always-yes?

The marvelous technique. Using the marvelous technique to move your team to 80% accuracy gives us this calculation:
                50% x (80% x $15 + 20% x $-2) + 50% x (80% x $0 + 20% x $-7)
which leads to $5.10. You don’t get to keep it all, though, because the marvelous technique costs $0.50 each time you use it, so you get $4.60 per decision. Then there’s that $500 up-front cost. The improvement over 60% is $4.60 – $2.70, or $1.90, so you can expect to break even on the up-front cost on the 264th improved decision. The improvement over always-yes is $4.60 – $4.00, or $0.60, which pays off on the 834th improved decision.

Uh oh. Until you find something better, the marvelous technique looks like a good deal. You give it to four randomly selected team members, Aaron, Abigail, Barry, and Bonnie. You match them with randomly selected teammates who, as a control group, will continue to use the current process.  Aaron and Abigail immediately strike gold. Barry and Bonnie immediately strike dirt. What’s going on? Is the marvelous technique not so marvelous? Does it not work for everyone? Are Barry and Bonnie sabotaging the change? Have Aaron and Abigail found a way to make it even better?

Streaks and slumps
To answer those questions, and to illustrate the answers, we’d find it helpful to have a decision simulator that can help us explore the costs and likely benefits of the current process and marvelous technique. Fortunately, it’s possible to build such a simulator. So I built one, basically a longitudinal Monte Carlo program. It uses the numbers and percentages above, combined with a random-number generator to simulate correctness in the appropriate proportions. We’ll use it here, doing something like the equivalent of clinical trials for a business decision. (You can have a copy. See below.)

Here are the initial results from Aaron and Abigail (the A team) and their control group. The chart covers their first 10 decisions. 

 Incidentally, there are slightly over a million possibilities for each of the lines. Some of those possibilities are much more likely than others.

The blue line is for the marvelous technique (the A team) and the red line is for the current process (their control group). The vertical axis is cumulative average performance. Here’s how we calculate it, using the blue (marvelous technique) line for illustration.

• In decision 1, the A team decided yes and the right answer was yes. They got $15, minus the $0.50 cost of using the marvelous technique, for $14.50.
• In decision 2, the A team decided yes and the right answer was yes again. They got another $14.50. Their cumulative total is $29.00, for a cumulative average of $14.50.
• In decision 3, the A’s decided no and the right answer was no. They got $0 minus $0.50. Their cumulative total becomes $28.50, for a cumulative average of $9.50. Notice that the line went down even though they maintained their perfect decision-making record.
• The only mistake made by the A team in the first 10 decisions is in number 8, where they incorrectly decided no. Their 90% hit rate is better than we’d expect, given the promised accuracy of 80% for the marvelous technique.

Their overall average, almost $10 for the first 10 decisions, is far above the predicted $4.60. That’s due not only to their better-than-expected 90% hit rate. It’s due also to there being 7 decisions where the right answer was yes, which pays $15 for a correct decision, as opposed to those where the right answer would be no, which pays $0 for a correct decision. Of course, the team doesn’t deserve credit for the attractiveness of the opportunities that were randomly presented to them, but it helps them look like a great team.

So, the A team did especially well due to a little luck on their decisions and quite a bit of luck on the nature of the decisions offered them.

Their control group, the red line, got 60% right, exactly as expected. They too benefited from the advantageous decisions: their mistakes mostly were of the $-2 nature (failing to take advantage of an opportunity) rather than the $-7 kind (investing in a bad deal).

Now let’s look at the first 10 decisions made by Barry and Bonnie and their control group.

A very different picture. The B team was right only 6 times out of 10, and they had fewer high-payoff opportunities. Their control group was right only 3 times. Although they’re below expectations the 60% and 30% scores are not unreasonable, given the percentages and the small sample of 10 decisions, but, combined with the poorer opportunities, the results sure look bad compared to the A team.

By the way, the differences between the two control groups suggests that the difference between the A and B teams is not entirely due to the teams’ competence or motivation. Yes, the A team got 90% right and the B team got 60%, but that wouldn’t explain the control-group results.

Incentive feedback
You’re a businessperson and you care about the bottom line. You see Aaron and Abigail’s 90% track record, versus Barry and Bonnie’s 60%; you see the A’s average of $9.80 per decision and the B’s average of $-0.30. You compare them to the targets of 80% right decisions and $4.60 per decision. To whom do you deliver a congratulatory pat on the back and to whom a motivational kick a little lower down?

The evaluation continues. Let’s look now at the first 100 decisions for the A team and its control group. We’re keeping the first 10; that is, the first 10 out of the 100 are identical to the chart above. Those 10 are scrunched to the left. 

Are Aaron and Abigail slacking off, and is their complacency infecting their control group? Their average results dropped by a third from the first 10 decisions to the end of 100. Maybe, you wonder, you were too generous with the back-pat. Well, at least they’re all still above the per-decision targets ($4.60 and $2.70).

What’s going on with Barry and Bonnie? 

They seem to be paying attention. What a turnaround! That lower-down kick obviously worked. They’re still below Aaron and Abigail, but they sure are improving.

The good news is that the marvelous technique seems to work for both teams, as they clearly and consistently outperform their control groups. It should, of course, since it’s right more often and the $0.50 incremental cost of using it isn’t very much relative to the potential benefit. However, the $500 one-time cost means that the A team’s marvelous technique is still $290 behind it’s control groups’ current process. The B team is only $210 behind its control group, not so much because the B team is so good but because its control group is so not so good.

Time goes on
Let’s go to 1,000 decisions. The first 100 decisions are scrunched to the left, and they are the same as we saw before. Here we see the A team’s marvelous technique (blue) and its control group’s current process (red). 

Both lines decline gradually from the highs after 100 decisions to numbers approaching our long-term expectations. The lines smooth out, of course, because we’re looking at cumulative averages. The results of any 10 decisions may be just as erratic as they were for the first 10 decisions.

If we were to show the chart for the B team and its control group, you would see that their lines are tilting up slightly.

As the early random noise dampens out with the improved perspective of 1,000 decisions, the marvelous technique pays off consistently. In fact, taking the one-time costs into account, it’s now a net positive of $1,315 for the A team (versus its control group) and $1,464 for the B team (versus its control group).

The end of the test
Finally, let’s see what happens after 10,000 decisions. The first 1,000 are scrunched to the left.

Here, again, is the A team and its control group. Except that this time, if you were to ignore the first 1,000 decisions, you’d find that its chart is almost identical to that of the B team and its control group. 

There’s still some noise from random gains and losses; that’s why both the blue line (marvelous technique) and the red line (current process) are so thick. Each thick line may contain long series of all-right and all-wrong decisions, which is to be expected.

And due to having a large sample that sorts out the random fluctuations, the A team’s long-term average performance — $4.55 per decision — is awfully close to B’s $4.57. Both essentially match the projected $4.60 and both beat the always-yes strategy, even after taking into account the one-time cost of the marvelous technique. The two control groups are within a dime of the $2.70 expected value.  And notice that there’s still room for improvement: as we calculated, perfect decision-making would improve this fictitious business’ results by more than 60%.

Decisions and distortions
We would conclude that the marvelous technique is a good idea. Notice, though, 1) we would have adopted the marvelous technique prematurely and unrealistically if we’d relied only on 1,000 decisions by the A team, and 2) we would have rejected the marvelous technique if we’d relied only on 1,000 decisions by the B team.

Moreover, the differences between the A and B teams were due 100% to random chance. There is no Aaron, Abigail, Barry, or Bonnie inside my computer; there is only a random-number generator, a series of simple equations as described above, and marvelously clever software that pulls it all together.

Rewarding the A’s looked ineffective, and punishing the B’s looked effective, because of how we interpreted the numbers, not because we actually motivated some electrons and pixels to behave well. (Notice, by the way, that if we’d taken a hands-off management style with the B’s, we’d have said that hands-off works. Ditto for rewarding their efforts, sending them for extra training, and so on. And a hands-off management style for the A’s would have looked just as ineffective as rewarding them.) The flaw in our interpretation is that we, as humans are prone to do, mistook correlation for causation.

I emphasize that I am not saying that performance is random. The most critical areas in the process we’ve just seen are those in which management and insight is involved, such as recognizing problems and opportunities, designing solutions, understanding costs, payoffs, and probabilities, ensuring that people are competent with the tools they use, and more. And just as randomness and analysis do not diminish the importance of management and insight, management and insight do not diminish the inevitability of randomness and the value of analysis.

Just how random can things get? Still using the numbers in our simulation, and based on 10,000 decisions, we see:

• Ten right decisions in a row seems pretty good, perhaps even streak-like. Well, with the marvelous technique’s 80% right decisions, the odds of getting 10 decisions right in a row are 10.7%. (Calculate it as 0.80 to the 10th power.) With 10,000 decisions, there are 9,991 opportunities (allowing overlaps) for 10-right runs. So, you would expect 1,073 such runs. The A team got 737; the B team had 882.
• With the current process’ 60% right decisions, the odds of 10 decisions right are 0.6%, so you’d expect 0.6% x 9,991, or 60 runs of 10 right in a row. The A team’s control group had 37 and the B team’s had 40.
• The odds of 10 wrong in a row are much lower because both strategies get most decisions right. For the current process we’d expect 0.01% of the 9,991 possibilities, or 1. We got 0 in both simulations. For the marvelous technique we’d expect a much smaller number, and we got 0.
• The always-yes strategy is right half of the time, since half of the time the right answer is yes. To get 10 right in a row, the odds are about 0.1%. (The same for 10 wrong in a row.) With 9,991 opportunities, we’d expect 10 streaks of 10 and 10 slumps of ten. The simulations ranged from 8 to 15.
• The longest run of right decisions came from the B team’s marvelous technique. It hit 38 in a row. The odds of that happening are 0.02%… which means we’d expect it to happen twice in this simulation. It’s a random streak, not a magic touch.
• The longest run of wrong decisions came from the always-yes strategy in the B team’s simulation. Its slump was 16 long. The odds were long — 0.0015% — but with 9,985 opportunities for a 16-wrong slump with 50/50 odds, it happened. The current-process control groups had a maximum slump of 9, and the worst marvelous-technique team got only 5 in a row wrong in its longest slump.

Conclusion
The point here is not that you should buy marvelous techniques, and the point is not that you have to wait for at least 1,000 decisions for a net benefit. After all, with different get-it-right percentages, payoffs, and costs we might see that a marvelous technique is a great deal, an awful deal, or much ado about not very much, and it might happen sooner and it might happen later. The point is not even that a single input number always separates the marvelous from the merely splendid or the potentially dismal, because all the factors must be considered together. (Not to mention that we humans are notoriously bad at running models in our heads. See When I Was Wrong.)

As our arithmetic demonstrated at the start of this essay, we don’t need a simulator to figure out the long-term average consequences of current processes and marvelous techniques. I ran many simulations in the course of writing this essay, and they all look similar near the end of the simulation (the thick lines) because they are averages of bigger and bigger samples. However, they are strikingly, startlingly, provocatively different at the start of the simulation… that is, at the time when we in real life would be judging the value of our decisions, and of our decision-makers. Use the simulator — you can have a copy, see below — to see for yourself.

The value of the simulation is in literally seeing (as in the charts above) what could happen, so we don’t mistakenly think that something is working when it isn’t, or vice versa. The value is in practical learning about the effects of randomness. As we saw, self-inflicted misinterpretations can lead us to reward or punish inappropriately and thereby pursue ineffective strategies.

The point of this essay isn’t that marvelous techniques are inherently profitable or that it takes a certain time for them to pay off. The point is the analytic process by which you judge them. The point is knowing what to expect if you shift from one decision-making process to another. The point is preventing a few oddball wins or losses from distorting your assessment. The point is thinking through your values for the input numbers. The point is the difference between the small-sample and large-sample runs. (Not unlike making sure that customer surveys, clinical trials, and political polls have large-enough samples.) The point is that a simple strategy (“always yes”) may perform better despite its mistakes than a more-expensive strategy that gets it right a little more often. The point is thinking rigorously and numerately. The point is how a discipline can help you learn when you are actually improving your odds.

If you’d like a free copy of the spreadsheet that produced the charts in this essay, please write to info@whatifyourstrategy.com. It allows you to enter your own numbers for your own analysis. The spreadsheet requires Microsoft Excel 2003 or later.

Scott Swigart of Cascade Insights (www.cascadeinsights.com) asked questions and made comments on the simulator that helped and inspired me. Thank you, Scott.

For further reading, see The Drunkard’s Walk by Leonard Mlodinow. [Update: Prof. Mlodinow has a terrific article in The Wall Street Journal, July 3, 2009: The Triumph of the Random.]

If you like the technique of 10 decisions, 100 decisions, 1,000, and 10,000, Google “powers of ten.”

Update. Low-scoring sports games can run into the same problem of randomly looking good or bad as we described above. See Richard Bookstaber’s “The Scoring Problem” on page W2 of The Wall Street Journal, July 10-11, 2010.

Carl Bialik (”The Numbers Guy” at the Wall Street Journal) published an article in the Journal on August 15, 2008, that echoes a numeracy theme we’ve long advocated at ACS: Extrapolating the past into the future is dicey. (See Precision In, Garbage Out and With All This Intelligence, Why Don’t We Have Better Strategies?) Extrapolations, says Mr. Bialik, would lead us to conclude that there will soon be more internet users than people. He writes also that straight-faced “studies” would project 100% of the American population will be obese by 2048.

So why do we extrapolate the past when it so clearly can be misleading to the point of absurdity? One possibility, of course, is simply that it’s easy, quick, and convenient; more-thoughtful approaches take more time and skill. Another is that it works well in typical spreadsheets (which spotlights one of many vulnerabilities in typical spreadsheets) and the social acceptance of everyone does it. Perhaps most of all it’s because it’s defensible. Defensible not because there can be more internet users than people. Defensible in the sense that the extrapolation will probably fit tomorrow, and the day after, and the day after that, even though we know that it won’t fit in the future (where “the future” is defined as “beyond the time I care about”).

We think extrapolation is self-evidently defensible, and it is merely on-the-surface defensible. It falls apart when things get interesting, which is precisely when we need help the most… and precisely when we are most vulnerable to bad advice.

Let’s extrapolate negatively — that is, look back in time — and recall the Y2K problem. That was the widespread computer problem caused by programmers who (for good, defensible reasons) coded years using two digits instead of four. When we rolled into the 21st century, their programs would think that we had gone -99 years ahead. It was a serious problem with potentially dire effects (lights-out was a mild possibility), and it took serious resources to forestall. Contrast Y2K and climate change. Y2K had a hard, specific, credible, non-negotiable deadline: December 31, 1999, 11:59:59 pm. Climate change does not. The thoughtful among us, notably former Vice President and Nobel Laureate Al Gore, urge us to create a deadline for ourselves.

We are lulled into passivity, on climate change and business thinking, by our constant observations that successive days look similar. We expect, from decades of daily reinforcement (shared by everyone else, who therefore don’t question us any more than we question them), that tomorrow will look like today, and the day after tomorrow will look like tomorrow. Tomorrow looks like today, and the day after tomorrow looks like tomorrow, but the day after tomorrow doesn’t look as much like today. The problem (stay with me here) is that tomorrow, not today, is when we will judge the day after tomorrow, and so it doesn’t look as different from today (when we started down the path) as it really is.

We rightly value breakthrough thinking. We value creativity, insight, and out-of-the-box thinking. Extrapolation does not lead to those goals.

Texans’ electric bills used to be among the cheapest in the USA. Now, six years after deregulation took effect with promises of lower rates, Texans pay some of the highest rates in the land.

Did deregulation fail? As you might expect, there is a lot of passionate commentary whizzing around the internet. Most of it is predictable. Some of it is intelligent.

But passion and comparisons of today’s rates to pre-deregulation rates don’t answer the question of whether deregulation worked. The answer needs to focus on one key word: otherwise. To know if something worked (or at least if it was a good decision, which is a different thing), we need to look at what would have happened otherwise.

Although we’re focusing on watts going on in Texas, the “otherwise” question is relevant whenever we want to know if something worked. Did the marketing campaign work? Did the mass-transit program work? Did the zoning laws work? Did the outsourcing work?

At one level we judge whether something worked by seeing if we achieved an objective. We wanted to double sales in two years, we ran the new marketing campaign, we doubled sales in two years. We leap to prior hoc ergo propter hoc, which is Latin for we ran the new marketing campaign, we doubled sales, therefore the new marketing campaign caused the doubled sales. “Therefore” is a heroic stretch. Maybe our new product or our new sales manager or our competitor’s stumble is responsible for our doubled sales. Maybe they did it together with the new marketing campaign. Maybe our sales doubled because we debottlenecked supply and distribution.

“Otherwise” talks about another twist, in which I get to use my remaining Latin, ceteris paribus, meaning all else being equal. Given the new product etc., maybe our sales would have stayed flat with the old marketing campaign, which would suggest that the new marketing campaign worked. Maybe our sales would have tripled if we’d had a different marketing campaign, which would suggest that the new marketing campaign failed (or at least that we failed in our selection of marketing campaigns).

Ceteris paribus isn’t easy. It’s the gold standard we use in double-blind clinical trials of drugs and medical treatments, but the business equivalent of clinical trials is difficult at best. So, we use technology to test otherwise. I’ve written elsewhere about the limitations in some kinds of analysis. In my experience, well-designed simulation is the way to go. (Incidentally, Skeptical Inquirer magazine reports in a review of Mark Buchanan’s The Social Atom that the state of Illinois successfully used simulations to avoid the missteps in California’s deregulation of the electricity market. ) But whatever the technology, the point is that it is possible to test otherwise.

Now, back to Texas. I understand that people may like or dislike the results (so far) of the deregulation decision and implementation, and, having read a newspaper article, I now have a passionate opinion. But preferences and passion, even my preferences and passion, don’t answer the did-deregulation-work question. Historical comparisons – electric rates then, electric rates now — also don’t answer it.

Preferences, passion, and history don’t tell us what Texas (and the rest of us facing knotty problems) should do next. What’s indispensible, the sine qua non (I guess I had one more Latin phrase in me), is that we must think clearly about the question and not leap to a satisfying-but-shaky answer. We need otherwise.

Update: Texas has announced that over several years it will more than triple the capacity of the parts of its electric grid needed to accommodate a major increase in wind power. Otherwise changes again.