by Mark Chussil

Albert Einstein said, “Things should be as simple as possible, but not simpler.”

No one wants to spend three months studying whether we ought to offer our new product in 182 colors or 183. No one wants to pluck decisions from the air when billions of dollars and thousands of livelihoods are at stake.

I didn’t say we don’t do those things. I just said no one wants to do those things. At least I hope no one does.

As simple as possible, but not simpler. So when it comes to business decision-making, what’s too simple, what’s not simple enough, and what’s just right?

Too Simple

I’ve been interviewed a few times recently (here, here, and here) about the value of daily-deal coupons from Groupon and LivingSocial. The deals are generally terrific for consumers. They’ve been good for Groupon and LivingSocial too. Groupon turned down a $6 billion takeover offer from Google early in 2011, and may be valuing itself in the $20 billion range. Not bad for a company that didn’t exist 5 years ago.

But are daily-deal coupons a good deal for the restaurateurs, merchants, and others who buy into the system and offer those deals to potential customers? How would you decide?

If you look around, such as in those interviews, you’ll find plenty of answers to that question. Some of those answers are learned, many of them are passionate, almost all of them are anecdotal. They’re not too simple because they’re wrong; on the contrary, they make good points. Rather, they’re wrong because they’re too simple.

They’re too simple because they isolate narrow pieces of the problem, like predicting life success on the basis of a high-school trigonometry test. (I hope that technique doesn’t work.) Yes, the coupons might cheapen a brand and that’s dangerous. Yes, on the other hand, the coupons might bring in new customers and that’s progress. But to make a good business decision about the coupons, we must account for both factors, and more.

In other words, the answer is it depends. Or, equivalently, there is no single answer that’s right for every business. Which is emphatically not the same as saying that there is no answer at all or that every business is different.

Before my interviews I spent a few hours creating a simple what-if simulator for restaurants that might use daily-deal coupons. It’s not pretty. Still, it handles multiple factors, such as how many tables the restaurant typically fills, how much the restaurant relies on repeat customers, and whether the restaurant has high fixed costs (e.g., fancy décor) or high variable costs (e.g., expensive ingredients). Based on those factors and others, you could get a reasonable idea whether the coupons would pay off for your restaurant. It depends. For some restaurants the coupons would pay off, for others they would hurt.

The point is that getting decision-making insight doesn’t have to be too complicated and that sound-bite answers are too simple.

Further reading: The Burden of Anecdote.

Not Simple Enough

With a friend and colleague I co-founded a company called Benefitics, which specializes in quantifying the social ROI of non-profit organizations. That means we estimate the monetary value to society of an organization’s activities and divide it by the cost of conducting those activities. We did it for Friends of the Children, on whose Board of Directors I am honored to serve, and for others.

We sloshed through a lot of academic research as we got our quantitative feet wet on our first project. We found deep thinkers whose studies shed rigorous light on genuinely tough social-science questions. The problem from our perspective was that their exacting methods were prohibitively expensive, in time as well as in money. Some studies had taken many years. If we modeled our analysis on those studies we would have to wait about 65 years for our results, which we felt was too long.

A great answer in 65 years would be useless (to us). Not simple enough. We didn’t need pinpoint precision or encyclopedic breadth. We needed something simpler: a good answer, soon. After all, an estimated benefit/cost ratio of 5.0 tells us what we really need to know even if the “right” number is 4.7 or 5.2.

So we developed a good answer. We used census data and government statistics rather than compile our own with new primary research. We made reasonable assumptions about what influenced what, and we tested those assumptions with smart people who know the field. Whenever we faced a choice about how to do something, we chose the analytically conservative path; if we introduced any bias, we would underestimate, not inflate, the results.

As thorough as a gold-standard scientific study? Our study was no slouch, but no, it was silver-standard. The good news: by avoiding not-simple-enough we saved 65 years.

Further reading: Precision In, Garbage Out.

Just Right

I ran a business war game recently in which the client and I wanted to use a simulation model so we could see which strategy ideas were promising and which were not.

We were budget- and time-constrained. I had one month to, among other things, put together a simulator. It had to be custom-calibrated for their industry. It had to handle multiple market segments, new competitors entering those segments, parallel universes, and the key “levers” each business could pull, to any degree, in any combination, and in real time. Impressed? I know I was, when it worked.

In the middle of the business war game strategists role-playing one of the client’s competitors took several actions including a price increase. The simulation model projected that the price-raising company would enjoy an increase in market share of several percentage points.

“Wait a minute!” cried someone from a competing team. “They raised their price and the model said their sales volume would go up? That doesn’t make sense.”

Was it a bug in a quickly developed model? Did the model fail to capture the way their industry worked? Did their industry have a perverse affection for higher prices?

No, no, and no. It wasn’t a bug; it didn’t misrepresent their industry; their industry, like most others, preferred lower prices.

Here’s what was going on. The price increase did not cause the gain in market share. It did not even contribute to it. The gain in market share came in spite of the price increase.

At the same time that they’d upped their price, they’d also spent more on marketing. Meanwhile, their competitors had moved, but less aggressively and less effectively. The gain in market share was due to the net interaction of everyone’s moves, aggressive or not, effective or not.

To demonstrate, we removed the price increase, left in all the other moves, and re-simulated. As you, I, and they would expect, the team would have gained more share if they had not increased their price.

And that’s what made the simulator just right on the simplicity scale. Sure, the model wasn’t “accurate.” (NB: no one can calculate “accurately,” or even discern accuracy, in any such analysis. But that’s another subject.) Even so, it moved in a sensible direction however the teams tuned their strategies, and the more forceful the action, the more the model would move. It gave sensible feedback. Just right for what we needed.

Further reading: What The Model Says.

Getting Your Analysis Just Right

Here’s what I’ve learned over 35 years working with analytic models. Well, not everything I’ve learned about them, but important stuff. And here’s what to notice and implement from those three simple stories.

An analysis is too simple if people, especially you, keep saying “but what if” or “but does it take into account” or “but that’s not realistic.” That’s the feeling you might have if you’re told Groupon is good, period, because it brings in new customers.

• It’s a bad sign when all the numbers in an analysis are about your business, when you’re never surprised by the results, or when you see happy numbers come out no matter what numbers you put in. It’s a bad sign when all you hear is anecdotes or when all you see is tunnel vision.
• It’s worth asking, as a sanity check on the analysis, if an big, obvious move by some¬one else (e.g., a competitor) would affect the analysis. If it wouldn’t, watch out.
• What may help it be just-right is adding realism. That may mean changing the conceptual underpinning of the analysis (e.g., don’t extrapolate history if you need to anticipate competitive dynamics). It may mean taking more factors into account.

An analysis is not simple enough if it won’t be ready on time to affect a decision, like a program that takes two days to calculate tomorrow’s weather forecast. That’s the feeling you might have when you find out that the data you really want will take 65 years to gather.

• It’s a bad sign when people quarrel about minute differences that just aren’t big enough to affect a decision. If that’s where the debate is, ask open-ended, non-confrontational questions such as what would we do differently if the minute differences turned out one way or turned out the other way.
• It’s worth asking, if the problem is that the analysis will take too long, if the decision can be delayed.
• What may help the analysis be just-right is to look for approximations or proxies. Approximations give you a reasonable number quickly. For example, you can average market-analysts’ forecasts of Microsoft Office sales rather than construct your own forecast. Proxies give you a similar or related number quickly. For example, look at sales of Windows PCs to get an idea of sales of Microsoft Office.

An analysis is just right if you can comfortably answer questions about and with the analysis. The analysis makes sense, it’s directionally correct, it focuses on the problem at hand. The analysis fits the need.

• It’s a good sign when people stop asking about the analysis and start getting excited about the results. It’s also a good sign when they want to run what-if experiments or have a copy to use themselves. Extra credit if they say they want to “play with” the model.
• It’s worth asking if you can apply what you created to related problems. The key is to recognize the general principles common to those problems. Experience at doing that is one reason I could built that strategy simulator in a month. (See also The Good, the Bad, and the Lucky.)
• What will help the analysis stay just-right is to filter proposed enhancements with this question: if we had this enhancement, what would we be able to do that we cannot do now? Which brings us to…

The Bottom Line

The purpose of an analysis is not to tell us a number we like. It’s not to deliver a definitive, immutable last word. It’s not to guarantee anything.

The purpose of an analysis is to positively affect the quality of a decision. Keep that in mind, and your analysis will be as simple as possible, and not simpler.

by Mark Chussil

I didn’t know that Connecticut, my home state, is so big! As reported by CNN.com, the not-yet-decided 2010 gubernatorial race stacks up with, so far, 50% of the vote for Malloy (D), 49% for Foley (R), and 12% for Horner (I). Wow, 111%. Then again, it doesn’t seem quite right that it took Horner 251,494 votes to earn that 12% while Malloy got his 50% with 565,508.

Oddly enough, CNN.com also reports that the undecided senatorial race in Washington has Murray (D) at  51%, Rossi (R) at 49%, and Horner (I) at 12% with 251,494 votes. 112%, even bigger than Connecticut.

Obviously there’s a bug. The details box that pops up retains the last independent candidate even in elections that had only a Democrat and a Republican. It’s not hard to figure out and it’s easy to fix. In fact it was fixed long before I finished writing this essay.

Then there are more-subtle bugs. Here we’ll look at people who switch, 20% liberals, a tremendous success rate, and customer satisfaction. What does that crowd have in common?

People who switch

“People who switched to ______ saved $$$!” Of course they did. They probably wouldn’t have switched if they weren’t going to save money, no matter how articulate the accented lizard or how perky the lady in the whitewashed store. The statement may be true but it’s not helpful. “Honey, it’s going to cost us $1,817 to switch our insurance, but those who switched saved lots so I think we ought to do it anyway.” If you hear that, don’t switch your insurance, switch your honey.

Okay, perhaps the people-who-switched pitch is relatively harmless. You’ll check your insurance premium against what you’re offered elsewhere before you decide, and presumably you’ll ensure that you’re comparing equivalent policies. The ads are designed to wake you out of your lethargy; after all, most of us buy insurance and then sleep on it.

But similar circular reasoning can lead people down questionable paths, especially if they’re depressed or thrilled. Such as after elections.

20% liberals

Pundits raked through vast stacks of numbers immediately after the 2010 midterm elections in the USA. The question was no longer who would occupy the high stations in government where they would soberly, solemnly, wisely, and selflessly help a great nation solve great problems, or where they would prepare for the 2012 elections if that seems more pressing. Once they knew who won and lost the pundits focused on why they won and lost. That answer can guide political leaders as they urgently seek to fulfill their promises to the American people, who had “spoken.”

One of my favorite pundits cited an exit poll in which people could describe themselves as conservative, moderate, or liberal. 20% said liberal. The pundit spoke at length with a guest about the size of that number. They debated whether people are reluctant to say they are “liberal,” since that term has been thoroughly tarred by conservatives, and discussed whether Democrats should more assiduously court moderates.

Not so fast. It was an exit poll. Exit polls are conducted on people who just voted. In 2010 conservatives voted in relatively great numbers and liberals in relatively small numbers; that, by definition, is why the Republicans did well. We may know that 20% of those who did vote consider themselves liberal, but we don’t know the percentage of liberals among people who could vote.

A more-useful question would be to look at the percentage of people in the general registered-voter public who describe themselves as liberal and contrast that to 20%. (Might also be useful to test the word “progressive” instead of “liberal.”) It makes a difference if Democrats lost because there are too few liberals or if they lost because there are plenty of liberals who didn’t go out to vote.

Tremendous success rate

There’s an organization that did better than the Republicans in 2010. I don’t know its name, and, to be honest, I’m not even certain it exists. I heard about it when I attended a seminar. But the story is good, and at least in theory such an organization ought to work.

The organization runs programs to help people make changes in their lives. On the first day of the program the facilitator tells participants to make a simple change in their homes that evening, such as reversing a roll of paper towels so that it feeds from the top (or bottom) instead of from the bottom (or top).

The next day, the facilitator asks who reversed their roll of paper towels. Those who left their paper towels unchanged are thanked, given a full refund, and politely asked to leave. The reason: if they’re unwilling to make a trivially simple change, they are not prepared to make bigger changes. They’re encouraged to return when they’re ready.

So the organization gets a tremendous success rate because they filter out those who do the equivalent of buying health-club memberships and never going to the gym. This clever approach may actually be a good thing because it quickly reinforces the behavior it seeks to instill, and it eliminates people whose attitudes might pull down the group. Still, it does not tell the potential customer what he or she really wants to know: what results will I get if I faithfully complete that program versus some other program.

Customer-satisfaction surveys

Having gotten this far, you may be able to predict the issue we’ll discuss regarding customer-satisfaction surveys. Take a moment to think about it.

Remember the exit polls as you think about it.

You do customer-satisfaction surveys with your customers.

Customer-satisfaction surveys can provide tremendously valuable insight into people’s experiences with a product or service. They don’t say much, though, about what made a person become a customer in the first place. To learn that, you need to contrast what was different about people who became customers (what did they want, what did they believe, etc.) from people who decided not to become customers. Ideally you’d split the latter group into those who decided to buy from your competitors and those who decided not to buy at all.

Here’s another way to think about it: customer satisfaction concerns the previous purchase, and for strategy and forecasting you need to worry about the next purchase. I was very happy with my trusty, powerful H-P calculator (I am dating myself, and all too accurately), and I’d give H-P fabulous customer-satisfaction scores. That said, I will not buy another calculator from them, or from anyone. I’ve got calculators in my computers, iPad, cell phone, and maybe in the smoothie maker I made famous in Motor Swilling Forbidden.

What they have in common

Switches, liberals, change, satisfaction. What do they have in common?

All concern people trying to use information to make better decisions. All are susceptible to circular reasoning (our customers love us, that’s why they’re our customers). All show how critical thinking can prevent mistakes.

• It doesn’t matter if those who switched insurance saved $450.22 or $453.87. It doesn’t matter if 20% of those who went to the polls were liberals, or 20.3%, or 19.735%. And so on. Whether the numbers we have are right is usually the smaller issue. The larger issue is whether we have the right numbers.
• It doesn’t take rocket science to avoid getting misled. Suggestion: state out loud exactly what a number represents and what’s missing, then see if it’s useful for you. When you hear “people who graduated from this program had tremendous success,” ask not only about the tremendous success but also about the people who. Who were they? How might they differ from you? How might they differ from those who did not graduate? How do they differ from those who did not enter the program?
• If you don’t have the right numbers but you’ve got something close, try to make an estimate. Getting close fast can let you make a reasonable decision fast. But beware: using shortcuts to get a reasonable number quickly is one thing, and using shortcuts to eliminate scenarios is quite another. See The How-Likely Case.

“USA Today has come out with a new survey — apparently three out of every four people make up 75% of the population.” — David Letterman

In related news

Kudos to Newsweek’s Sharon Begley, for “Wanted: BS Detectors” (“bs” = “bad science”), in the October 28, 2010, issue of the magazine. “Understanding what counts as evidence should … trump memorizing the structural formulas for alkanes.”

Update, April 14, 2011. “Tasting Wine: A Coin Toss?“, from the Republic of Math. A random person might correctly judge whether a wine is swell or swill with coin-toss odds. That’s not the same thing as saying that a given person has only coin-toss odds of judging multiple wines correctly.

Update, January 5, 2012. Kudos to The Wall Street Journal’s Carl Bialik, “The Numbers Guy,” for “A World Full of Alarming Traces.” He discusses why the finding that 9 out of 10 baby-changing (odd term, “changing” your baby) stations test positive for cocaine powder, while not a good thing, may not be as alarming as it sounds.

[This essay was published by SCIP in the October/December 2010 issue of Competitive Intelligence Magazine.]

Good judgment comes from experience, and a lot of that comes from bad judgment. — Will Rogers

What will you experience in this essay? I don’t know. Different people read the same words and see different stories. Which, actually, is what this essay is about. Which story should we believe?

Experience, CEOs, and profits

We watch Hollywood CEOs dispense orders while they march down endless hallways with the top three levels of their organization charts trailing in strict hierarchical order. All hail the CEOs and bend to their wills. Within minutes bottom lines surge and fortunes are made.

We watch real-life CEOs being shamed, blamed, and flamed. Off with their golden parachutes! Help wanted: new blood. Who wants to go next?

A sensible question, and not so easy to answer. Who should go next? An insider who knows the business or an outsider with a fresh perspective? A young hotshot or a seasoned veteran? Are real-life CEOs destined for front-page failure? Can we get a Hollywood CEO from central casting?

Professors Monika Hamori of IE Business School in Madrid and Burak Koyuncu of Rouen Business School in Rouen, France looked into such questions. The Wall Street Journal wrote about their research on October 3, 2010, in “CEO Redux Not Always a Hit.”

Of course, pretty much nothing is always a hit. For that matter, pretty much nothing is always a miss. That headline is as informative as “coin doesn’t always turn up heads.” But you and I assembled here to quibble over conclusions, not headlines. The conclusions over which we will quibble are those reported by the WSJ, since I was unable to find the original research paper with the professors’ conclusions.

CEOs gone bad, it seems

Here are the first two sentences from the article:

“For chief executives, past experience doesn’t necessarily lead to future success.

“A new study found that CEOs who have previously held other CEO posts actually perform worse than people who have never been CEO, judging by a key metric.”

What message do you get from those lines and from the “CEO Redux Not Always A Hit” headline? Probably that experience is overrated. Such a conclusion would be newsworthy indeed, which is presumably why it turned up in the news.

The article went on to cite supporting data: CEOs with prior experience as CEOs averaged 3.92% (such precision!) return on assets over three years, and first-time CEOs averaged 5.4% ROA. Repeat CEOs who stayed in the same industry averaged 3.1%. Repeat CEOs who stayed with similar-sized companies averaged 2.94%.

It doesn’t sound good for experience. No one wants to give up a point or two of ROA, not even a CEO. Still, we’re not ready yet to reach for conclusions, let alone accept the implication that a company should not hire as CEO anyone who has previously been a CEO.

Introductory quibble: does it matter?

The WSJ article does not say whether the differences in ROA are statistically significant. Even if they are, we can quibble whether the differences are managerially meaningful. Considering the tremendous variation in ROA among companies, the difference between 3.92% (or even 2.94%) and 5.4% is not very big. That narrow range would suggest that, as effects go, CEO experience is minor.

More important, three-year average ROA is a dubious metric for answering the experience question. For a better metric, see the next quibble.

Important quibble: who’s getting better?

We don’t know from the WSJ article whether there were differences in ROA in companies before they got the first-time or repeat CEOs. Perhaps the 3.92% average ROA for repeat CEOs was an improvement from a lower number. Perhaps the 5.4% ROA for first-time CEOs was a reduction from a higher number. We should be looking at the change in ROA that happened after hiring new CEOs. If ROAs went up after hiring experienced CEOs or went down after hiring first-time CEOs, we might look more kindly on experience. Plus, see the next quibble.

Most important quibble: cause and effect

Now we’re going to quibble with our sleeves rolled up. There are other plausible explanations that might explain the differences in ROA, and the data reported in the WSJ article do not rule them out.

Say you’re an experienced CEO. What kind of company might be especially eager to hire you? One that’s in trouble — e.g., low ROA — and wants to improve its performance. That would imply that low ROAs “cause” hiring experienced CEOs, as opposed to experienced CEOs causing low ROAs.

Or say you represent the board of directors of a highly profitable company. Are you more likely to bring in a CEO from the outside (by definition, a repeat CEO) or to promote from within? I think the insider, who presumably knows how to keep the good times rolling. Say you’re the board for an unprofitable company. Do you want the outsider or the insider? I think the outsider, because you want changes made. Net effect: high-ROA companies with first-time CEOs promoted from within to stay the course, and low-ROA companies with repeat CEOs called in to perform a rescue.

Although those explanations make sense to me, I don’t know if they are true or if the professors conducting the research have data that prove or disprove them. And the WSJ article cited a few other ideas, such as repeat CEOs thinking outside the box (score one for experience) but not being familiar with what works in the company or industry (score one against experience).

The question in reverse

Still, all quibbling aside, the very idea that experience may not be inherently valuable is intriguing. (Which I admit with some reluctance, as I have spent a lifetime accumulating experience. Of course, so have you.) We might ask the question in reverse: why is it even possible that CEOs without experience might outperform CEOs with experience?

There are the hypotheses mentioned in the WSJ article. There are others, most notably about possible overconfidence in experienced CEOs (contributed by Seena Sharp), that I’ve seen on the LinkedIn group for Strategic and Competitive Intelligence Professionals.

Hypotheses are nourishing to the brain, so I’ll add yet another. I think we have all seen people make different use of their experiences:

• Some people seem to emphasize longevity and achievement when they mention experience. “You should believe me because I have survived many years in this field and achieved success.” (See also It’s Working!)
• Others seem to emphasize learning. “You should believe me because I have observed and thought deeply.” Note that learning doesn’t mean or require a classroom.

We’d have to split the CEOs in the study into those groups to see if experience-style makes a difference. I’d hypothesize that it would, that CEOs (and others) who rely on longevity or achievement would underperform those who rely on learning. That’s because longevity and achievement alone merely reinforce previously acquired ideas or behavior, and it takes observation and thinking, which take time, to learn. But until those data are in, all I’ve got is a hypothesis, not facts.

“I don’t think much of a man who is not wiser today than he was yesterday.” — Abraham Lincoln

Sadly lacking newsworthiness

So, working solely from what’s presented in the article, what we have is a questionable headline based on simple statistics that may or may not be significant, meaningful, or properly interpreted in the article. Our conclusion, sadly lacking newsworthiness: we don’t know whether experience is the culprit, the victim, or the statistical equivalent of an innocent bystander.

Chalk it up to critical-thinking experience.

Nothing is a waste of time if you use the experience wisely. — Auguste Rodin

Preamble.

This is the third and last in a series of essays about models and competitive strategy.

From All About Models, the first in the series:

• Models describe how we believe the world works.
• Models may be in our heads (mental models) or computer-based.
• Computer-based models are always based on mental models, though they can calculate more and better.
• A model is always involved, though perhaps unconsciously, when we say or compute if we do this we will get that.
• No model is perfect. The test for whether a model is useful is whether we can make a better decision with it than without it.

From What The Model Says, the second in the series:

• What the model “says” is at least as much about its design as about the numbers that come out. For example, a model that knows about costs will give you cost-based advice, nothing more.
• The most important decision we make about our model is to choose how our model should think.
• If your model omits key variables, it will leave you vulnerable to competitors who make use of those variables.
• A key test for whether a model is valid is whether it makes conceptual sense for the problem you want to solve. Many common modeling techniques do not.

Excessively cautious notice. This essay may occasionally sound commercial because in it I describe techniques ACS uses, including one technique proprietary to ACS. The two business war-gaming techniques I’ll describe are practiced by ACS and other companies; each company has its own way of doing things. The third, strategy decision tests, is proprietary to ACS. Related techniques (e.g., Monte Carlo simulation) are not.

End of preamble.

People whisper to horses, dogs, and (according to TV) ghosts. Why not models? The model whisperer — perhaps you, savvy strategist — gently, wisely guides models into shape and helps them achieve fulfillment as oracles of your business’ future.

In our last episode I promised that we’d explore when to use mental and computer-based models in competitive strategy. I further promised that we’d talk about them in the context of three strategy-development techniques: qualitative business war games, quantitative business war games, and strategy decision tests.

In qualitative business war games we experiment with ideas and paradigms embedded in our mental models. In quantitative business war games we use computer-based models to stress-test specific strategy options. In strategy decision tests we run massive what-if simulations to gauge the risks and rewards of going down different paths.

The hypothetical examples I’ll present are actually not hypothetical. Each is an amalgam of multiple war games I’ve conducted in multiple industries, with identifying marks removed to protect confidentiality.

Qualitative business war games

Imagine. You run a business preparing a major change. You’re contemplating something unprecedented, even controversial. Your move will be highly visible and very expensive to retract. If it works, the rewards are great. But there are many interested parties out there, including your competitors, and it’s far from clear which side they’ll take. Their opposition could hurt you badly. Should you make the move?

You’ve thought it through and, although you favor making the move (your mental model at work), you’ve quickly bumped up against the chess-like complexity of your situation. You feel how easy it is to slip into rosy predictions. Your management team is split, some offering passionate support, some passionate opposition (their mental models at work). You and they have debated for some time without reaching a conclusion. Your window of opportunity is closing.

What would you do? Why, you’d run a qualitative business war game, of course.

Business war games involve role-playing. Teams of strategists role-play their business, competitors’ businesses, government regulators, consumers, investors, and so on, as needed for the problem the company wants to solve. Through a series of structured exercises and interactions, the teams construct, debate, select, execute, and react to strategy moves.

In the safe environment of a qualitative war game your strategists get to experience what could happen as they roll out a strategy… or, as a role-played competitor, as they fight against it. A qualitative war game is not unlike a mock trial or a field exercise: it doesn’t ascertain what will happen, but it shows what could happen, with far greater richness than a conference-room discussion. It reveals gaps in your thinking. It surfaces unconscious and unfounded assumptions that could haunt you later.

A qualitative business war game plays out your mental model of what you believe or hope will happen. In addition, it will probably uncover new ideas, favorable and unfavorable. You may choose to explore those new ideas with the quantitative methods I’ll describe later, or by running through another qualitative session. It’s because of those new ideas that I strongly advise you to run at least two “rounds” in a business war game; the second round turns the clock back to the kick-off point of the war game, but with the first round behind them the teams know more than they did before.

By definition, qualitative business war games don’t require computer-based models. They are flexible, time- and cost-efficient, and they can simulate just about anything. They are a great way to generate ideas because the adrenaline rush (I’m not kidding) of simulated competition stimulates creativity. Oh, and they’re fun.

Their drawbacks mirror those of mental models: humans make lousy calculators, even more so under uncertainty, and so it is difficult to get reasonable performance projections in a qualitative war game. Frankly, I don’t even try. I wouldn’t believe the projections.

Quantitative business war games

Imagine. You run a business facing a new competitive threat that’s sure to materialize. You and your colleagues have developed several mutually exclusive countermoves. The countermoves share no common ground and there’s no split-the-difference compromise to be found. Your colleagues advance persuasive arguments in favor of each possible move, and promise future glory. But how can that be, when the moves are so different? How can you know what to expect? What if competitors counter your countermove?

You ponder and reject the usual tools. Financial spreadsheet: doesn’t take competitors into account. (Notice the mental model in those spreadsheets, and refer to our discussion of Model V and Model M in What The Model Says.) Forecast: history won’t be a good guide because the new competitive threat changes too much. (Notice the mental model there too.) SWOT analysis: just a way to express your mental models using four categories.

So you turn to a quantitative business war game. You use a qualitative game when you’re looking for ideas and surprises; you use a quantitative game when you’re looking for analysis and choices. A quantitative war game uses teams and role-playing like the qualitative variety, but it adds (as you might expect) a computer-based model. The model, calibrated for your competitive environment, takes as input the strategy decisions made by the teams and provides as output estimates of sales, market share, profits, and so on.

Sidebar. Tempting and fascinating as it is, I’m not going to discuss the process of building such models in this essay. Contact me at info@whatifyourstrategy.com if you’d like to talk about it. I’ll just say 1) it’s possible to build such models, 2) it’s not as hard as it might sound, and 3) such models easily satisfy the usefulness criterion, to wit, whether you can make a better decision with the model than without it. See, for example, the Shell case at the start of Putting the Lesson before the Test from Wharton on Dynamic Competitive Strategy (Day and Reibstein, editors, 1997). That model projected a nine-figure loss if Shell were to execute a certain strategy. End of sidebar.

In qualitative war games participants are surprised by the actions teams take. In quantitative war games participants are surprised by the outcomes of those actions. Surprises are good, of course, if they’re in the safe environment of a war game.

Here’s a tip-of-the-iceberg illustration that shows the dynamic of a quantitative war game. Teams develop strategies. They all expect to perform well. Unless it begins with a near-monopoly, no team expects to lose market share. Yet each and every market in the world always, always contains exactly 100% market share, which means that for one business to gain share someone must lose share. Without a model, the teams argue endlessly about who gains and who loses. With a model, teams see who gains and who loses, and why. It makes the discussion much more fruitful.

By definition, quantitative business war games require computer-based models. They generate ideas, as do qualitative games, but their key benefit is that they let you evaluate and contrast your strategy options rigorously and objectively. They can produce consensus and action quickly because participants see the results of option A versus option B. (I’ve seen Fortune 500 companies change course overnight.) They help set reasonable targets and expectations. Oh, and they’re fun.

Their drawbacks mirror those of computer-based models: more time and cost to set up, and the model won’t be able to simulate actions not anticipated in the design phase. More subtly, some models are simply better-designed than others, so you must choose carefully.

Strategy decision tests

Imagine. You direct pricing for a business in a highly competitive market. You and your competitors watch each other’s prices like genetically modified hawks. You and they can change prices frequently. How should you set your prices?

This situation is rife with mental models and computer-based models. Some pricing strategists — your competitors? — look at prices periodically and then make a decision what to do. Those decisions, of course, come from some invisible and visceral combination of their mental models and any analysis they see. Other pricing strategists — your competitors? — may use computer-based models that figure out “optimal” pricing based on, for example, demand curves or the status of supply and demand at this microsecond.

So how should you set your prices? This much is clear: if you go too low you may trigger a price war or leave money on the table, and if you go too high you may lose share and long-time customers. If you try to duplicate others’ apparently successful moves you may make matters worse, sort of like grabbing someone else in midair instead of opening your own parachute. And of course prices are relative, and of course your prices affect their prices, and vice versa. So how should you set your prices?

You set up a strategy decision test.

Sidebar. Or perhaps you use other tools, of which there are at least several. As my focus in this essay is models and decision-making, not a pricing-technique critique, I won’t delve into methods other than strategy decision tests. I’ll just reiterate that all computer-based models are based on mental models, and it behooves the model consumer to ensure that a model is conceptually sensible and appropriate for the intended use. End of sidebar.

A strategy decision test is a massive what-if simulation. It’s useful when there are many options for you and/or your competitors and you want to understand the risks and rewards of your options. Each of your options may face many thousands of possible outcomes, and evaluating all your options may mean looking at many millions of futures. (See also The How-Likely Case.) That’s too many to do in your head. It’s also too many to do in a spreadsheet. But it’s not too many for a strategy decision test.

Strategy decision tests look at all the possibilities and summarize the results. They don’t involve teams or role-playing. They do, however, thrive on ideas for strategy options, specifically including ideas generated in war games or by brainstorming.

Obviously, strategy decision tests use computer-based models, and all my usual imprecations about model quality and sensibility apply.

One of the benefits of strategy decision tests is its comprehensiveness, which lets it show a clear view of risk; that is, of the range of what could happen. That comprehensiveness would have been very difficult not many years ago, and simply impossible not many years before that.

Another, quite intriguing, is what Malcolm Gladwell called “serendipity” in his article about drug companies screening millions of compounds to see if they might have useful medicinal effects. (“The Treatment,” The New Yorker, May 17, 2010.) “[Screening] provided a chance of stumbling across something by accident — something so novel and unexpected that no scientist would have dreamed it up. It provided for serendipity, and the history of drug discovery is full of stories of serendipity.” He cites penicillin and Viagra. “What he [a cancer researcher] found was exactly what he’d hoped for when he started his hunt: something he could never have imagined on his own.”

That serendipity happens in strategy decision tests. It’s not only that I’ve seen strategy decision tests find strategies that beat the best I could develop. I’ve also seen strategies pop out that I could never have imagined on my own.

The drawbacks of strategy decision tests are two. First, developing a strategy decision test takes somewhat longer than developing other kinds of computer-based models. Second, the analysis may appear like a black box, not because anything is hidden but because the scope is too big to be eyeballed. Oh, and maybe a third. The results may be fascinating, insightful, and even jaw-dropping — I’ve felt mine fall a few times — but it’s not as entertaining as a business war game.

The n rules for model whispering

My goodness, we made it! Thank you, friendly reader, for staying with me to the end (almost) of this series of long essays.

We end with a few rules to remember as you whisper to your models and make your competitive-strategy decisions.

There is always a model. Computer-based or mental; Model V or Model M or Model XYZ; big or small. You never decide whether to use a model. You always decide which model to use. Don’t expect a financial spreadsheet to give you advice about marketing strategy. Corollary: choose a model on the basis of sensibility for the decision you need to make.

Computer-based models are people too, sort of. Computers only know what people have fed them and they only think what people have taught them. It’s not human versus computer. It’s, again, which model you want to use, and is that model better applied in your head or in a computer. Corollary: don’t try to do too much arithmetic in your head.

Models can stimulate creativity. People get results from models — their mental models, others’ mental models, computer-based models — and they create ideas for how to do better. (War games are a prime example.) They even get ideas by designing models, as they become attuned to the wide range of variables under their control. Corollary: you can build your creativity by asking yourself “how would I model that?” when you come across an interesting strategy. That activity thrills me and makes me endlessly fascinating at parties. I know that because people can hardly wait to wander off and think about it on their own.

Surprise is good. If you can always predict what a model will say, the model doesn’t add any value. The model adds value when it tells you something you didn’t already know. Corollary: if you (i.e., your mental model) disagree with another model, take the opportunity to learn whether to update your mental model or to jettison the other model.

There is no such thing as precision or accuracy about the future. There are many, many, many possible futures. No practical model can capture every relevant variable and tell you which one will happen. (Put another way: a perfect model of the next five years would take five years to run.) Depending on the technology we employ, you can get a good / better / best view of those futures, but no one can guarantee a correct view. Incidentally, there is also no such thing as data about the future, but that’s only one reason why there’s no such thing as precision or accuracy about the future. Corollary: the objective is to improve the odds of making good strategy decisions.

Finally: ask what-if, and imagine. “Whenever you see a successful business, someone once made a courageous decision.” — Peter Drucker

Further Reading

About business war games in general
Business War Games
Learning Faster Than The Competition
The Seven Deadly Sins of Business War Games

About quantitative business war games and methods
Honey, We Shrunk The Industry
Honey, We Shrunk The Industry Again
Precision In, Garbage Out

About strategy decision tests
Predicting Competitors
Strategy decision tests
When I Was Wrong

Preamble. This essay is the sequel to All About Models. In that essay we covered several points:

• Models describe how we believe the world works.
• Models may be in our heads (mental models) or computer-based.
• Computer-based models are always based on mental models, though they can calculate more and better.
• A model is always involved, though perhaps unconsciously, when we say or compute if we do this we will get that.
• No model is perfect. The test for whether a model is useful is whether we can make a better decision with it than without it.

You don’t have to read that essay before this one, but it wouldn’t hurt. End of preamble.

So you’ve got a model, whether mental or computer-based. Or several models, put forth by several people or several computers. Congratulations! Your brainstorming, calculating, visioning, calibrating, and/or programming has paid off. Now you’re ready to get value and insight.

So, what do you do next? How do you get that value and insight?

Not so fast. It’s not so easy. In fact, you may be shocked a few paragraphs from now.

Before we listen to what a model says, we must figure out whether it speaks well-founded wisdom or well-intentioned folly.

Model V versus Model M

In All About Models I posited two models for a large manufacturing business. Both models acknowledged that our business has high fixed costs and that we must cover our costs to be profitable. They diverged from that point.

• One model said costs per unit go down as we produce and sell more units, and that selling more units also brings in additional revenue. We’ll call that Model V, for volume.
• The other model said it’s easier to cover our costs if our prices are high, and that excellent quality and a strong brand give us pricing power. We’ll call that one Model M, for margin.

Model V and Model M could be implemented as computer-based models or remain as mental models. It doesn’t matter right now.

For Model V, “what the model says” is to increase volume. For Model M, “what the model says” is to increase margin. Notice:

By listening to what the model “says,” we listen to the model’s design
at least as much as we listen to the numbers sifting through
our mental or silicon calculators.

We made the most important decision of all when we built the model: we chose which way, volume or margin, is the correct way to think about (i.e., to model) our large manufacturing business. Any subsequent analysis and action issues from that chosen paradigm. If we’ve chosen Model V, for instance, and if our profits underperform, we will look for other ways to boost volume. Why volume? Because the model we adopted, which codified the way we thought, shows us that volume drives profits.

The power to speak

What we hear from the model depends on which model we’ve given the power to speak. Choosing the wrong model messes up a lot, perhaps even everything. (See also Predictable Competitors and Predicting Competitors.)

In this example, neither Model V nor Model M is right. They’re not right because each misses something important, namely, what’s in the other model!

And that’s not all. For example, neither takes competitors’ actions and reactions into account. Sure, your unit sales may go up if you cut your price. But what if competitors cut their prices too? Okay, perhaps total demand will grow. Will it grow enough to compensate for the lower prices? Maybe yes, maybe no. The point is that we won’t even get to that question if our model ignores competitive dynamics.

The situation isn’t necessarily quite as dire as it sounds. We can take dubious, indulgent comfort in the notion that our competitors may make the same mistakes we do. If a whole industry adopts a way of thinking (and many industries do), no single business suffers a competitive disadvantage due to choosing the wrong model. On the other hand, a business could potentially enjoy the benefits of a different model if it can find one. That’s what those upstarts coming out of left field do. That’s what we call “game changing” action. Changing the game means changing our models and starts by changing our minds.

This discussion of Models V and M is highly and vitally practical. But rather than lengthen a series of long essays by reciting my evidence, I’ll just provide a link to real-life war stories. (Look about halfway down that post.) See also the story of Shell that begins Putting the Lesson Before the Test, a chapter from Wharton on Dynamic Competitive Strategy (Day and Reibstein, editors, 1997).

Does it make sense?

There is always a model when we estimate, assert, or calculate that this action will lead to that result.

The challenge in developing an effective strategy begins long before we decide what actions to take and how to execute them. The challenge begins when we decide what model to use. By selecting a model we frame the way we see and evaluate our strategy options.

Prudent strategists want to make sure a model is valid before they select it and thereby entrust their businesses and their careers to its advice. Based on what we’ve discussed about Models V and M, I suggest that “valid” refers first and foremost to whether a model makes sense.

For example:

• A model that looks only at price does not make sense if customers perceive differences in product or service quality.
• A model that calculates changes in market share does not make sense if it ignores customer loyalty, inertia, and switching costs.
• A model that accounts only for our business’ actions does not make sense if we have competitors.
• A model that extrapolates the past into the future does not make sense if we expect (or want) the future to look different from the past.

And so on. Those statements reflect what ACS calls principles of competition.

Three things to notice about that brief “a model that” list. First, none of those statements is particularly controversial. Second, none of those statements is particularly complicated or arcane. Third, according to those statements many of the models commonly used in strategy development violate the does-it-make-sense criterion.

Sensible models are difficult to handle in our heads, partly because of the arithmetic and partly because of the number of we do this, then that happens, then that, then that, then… connections. That’s why I favor computer-based models for many applications. There are times, though, when computer-based models are overkill or even a distraction.

In my next and final essay in this series, The Model Whisperer, we’ll explore when to use mental and computer-based models. We’ll apply those models in qualitative business war games, quantitative business war games, and strategy decision tests (sometimes called decision tournaments).

This essay is full of good stuff about models. Of course that depends on how you define good stuff and models. If you mean something involving delectable (your definition here too) folk lurching down a runway with sour expressions stitched onto their faces, well, sorry, I’m not that kind of manager. But if “good stuff about models” to you means cool and insightful methods to improve decision-making, as it does to me during working hours, then you’ve clicked to the right place.

The fascinating world of models and decision-making

All of us live with models. More accurately, models live within all of us. They help us every day.

As if that weren’t enough, there are models outside us too, and they help us too.

In general, we can classify models chemically: those based on carbon and those based on silicon. The former, mental models, operate inside our heads. The latter, computer-based models, operate inside computers. They behave differently.

Mental models

Mental models are the things in our heads that say if you do this then the result will be that. They reflect knowledge and/or beliefs about how the world works.

Our ability to, say, throw physical objects to a specific target without consciously solving equations comes from mental models. If asked, we could enumerate the key variables involved — the force we apply, the distance it must travel, wind resistance — but we don’t think about them when we throw an object.

Those mental models work well, and without controversy, because thrown objects precisely obey the laws of physics for everyone, every time, so far. But not everything covered by mental models is so consistent. Humans’ mental models can differ dramatically and contentiously where our experience is more varied, complex, or ambiguous. As with competitive strategy.

Imagine you and I run a large manufacturing business. A mental model for what makes it tick may go like this:

• We have heavy fixed costs.
• To be profitable, we must cover those fixed costs and all other costs.
• Fixed costs go down per-unit if we sell more units.
• Selling more units also brings in more revenue.

I’m not saying that mental model is right or wrong. I’m saying only that it is an example of a mental model.

That mental model reflects a certain list of variables (akin to those for throwing physical objects) and it leads its owner to think in a particular way. It focuses on internal operations (our costs), it recognizes the effects of volume (keeping the factory full reduces costs per unit), and it would favor moves such as price cuts or mass marketing because, given the list of variables, those actions would promise to build volume.

I’ve encountered exactly that mental model in the business war games I’ve conducted, and I’m sure you’ve heard it too. It’s common and it at least appears to make sense.

Think again about our large manufacturing business. What would be a different mental model? How about this:

• We have heavy fixed costs.
• To be profitable, we must cover those fixed costs and all other costs.
• It’s easier to cover our costs if our prices are high.
• Excellent quality and a strong brand give us pricing power.

I’m not judging that mental model either. Notice, though, that it reflects a different list of variables and it leads its owner to think in a different particular way. It focuses on customer perceptions (quality and brand), it recognizes the causes of margin (profit comes from the difference between prices and costs), and it would favor moves such as price increases or product differentiation because, given the list of variables,  those actions would promise to build margin.

I’ve encountered that mental model as well, and I’m sure you have too. It’s common and it at least appears to make sense.

Our large manufacturing business is hurting. What should we do? Your mental model says cut price, my mental model says don’t. We battle, we spreadsheet, we PowerPoint. We argue passionately about why a price cut is a good idea or a bad idea, we selectively cite anecdotes and trends and studies. Oddly, we spend less time asking how we can end at different conclusions when we start with the same data; that is, asking how our mental models differ. (See also Predicting Competitors.)

The point is not that any given mental model is right or wrong. (That said, we all know that no one is always right.) The point isn’t even that we must eventually choose one mental model or another. The point is that we all have mental models, we use them all the time, they’re unconscious to us and invisible to others, they appear self-evidently true to their owners, and they differ from person to person.

Mental models have benefits.

• They’re as flexible and creative as the human brain.
• They accommodate new information in real time. We can revise or update them quickly; we just change our minds.
• We always have them with us, they’re always ready to go, and we don’t need help from IT to operate them.
• Mental-model conflicts, annoying as they may be, help us learn. The wisdom of crowds. The marketplace of ideas. Very beneficial in qualitative business war games.

Mental models have drawbacks. For example, they require that we handle numbers in our heads. Sure, we can handle the math behind accurately throwing objects, but try this much simpler arithmetic: what’s 1,248 x 3,579 – 9,876 + 5,555? You’re right, it’s 4,362,271. I was just checking to be sure you knew too. How about this: if our sales are $1,000 today and we expect 3.15% compound annual growth, what will our sales be after 63 months? And we haven’t even gotten into any interesting problems yet. How about our profits after 5 years, assuming that brand popularity rises 9% and fixed costs are… well, you get the point.

It’s not just about handling numbers; it’s also about understanding ripple effects. We cut our price and expect volume to grow, but our competitors want volume too and they feel threatened and so they cut their price in response, so now we’re still at parity but at lower prices, so we cut our price again or cut our costs, and then, and then, and then. Unintended consequences come from unanticipated effects. Scenario-planning and role-playing programs, like business war games, help reveal those effects and consequences. (See also Do Not Overtighten.)

For an amusing and related diversion — and we probably all need one right about now — you might enjoy the lyrics to Tom Lehrer’s classic satire New Math.

Computer-Based Models

Remember that our mental models tell us if you do this then the result will be that. Say you tell about computer about “this” and ask it what “that” will be. Sorry; it will just sit there in stolid silicon silence. Before the computer can tell you about “that,” you have to tell it how. Your computer thinks as you tell it to. Your computer thinks like you.

All computer-based models start in life as mental models.

If you believe you should forecast your sales by extrapolating trend lines into the future, you or your Excelophilic proxy will insert trend-line-extrapolation equations. If you believe you  should forecast sales with a statistical model, you will commission a statistical model. If you ask someone else to figure it out, your computer will think like the person to whom you delegated thinking. And if you believe your mental model is good enough, you won’t even turn on your computer. Let it sit there.

Computer-based models think neither better nor worse than humans. That’s because they think like humans. Their benefits are in clarity, speed, precision, and scope.

• They’re explicit and visible. People can see, discuss, enhance, and ultimately share the way of thinking embodied in the model.
• They compute far, far faster than humans can. That’s more than a convenience. It means also that it’s practical to conduct serious what-if tests with many scenarios. A simulator I wrote calculates about 20,000 scenarios per second, which is handy when I ask it to process millions. (See Millions of Pricing Simulations.)
• They calculate much more accurately than humans can. (See also Precision In, Garbage Out.) Oh, by the way: the arithmetic we covered earlier? The real answer is 4,462,271. The 4,362,271 I mentioned was a sly test. You didn’t know it was wrong? That’s the point.
• They can keep track of many more variables than humans can. A strategy simulator I created for a company could handle roughly a thousand decisions for them and their competitors in multiple countries. Not a job for a mental model. Very beneficial in quantitative business war games and strategy analysis.

Computer-based models also provide a different kind of benefit. Yes, they serve as mental models mated with deft calculators. They also let strategists compare mental models. Back to our large manufacturing business, the one where you and I have been arguing whether to cut price. Use a computer-based model to run it both ways. And maybe some other ways too: cut capacity, broaden the product line, lead prices up, and more.

In other words, computer-based models allow us to run what-if tests. They can do so in quantitative business war games or even in the massive explorations ACS calls decision tournaments. (See also When I Was Wrong.)

Computer-based models sometimes are like rocket science, but their drawbacks are simple. They take time, money, and skill to create. And, of course, they need mental models for their raw material, which makes them vulnerable to lousy mental model in, garbage out. Note that the garbage-out would be the fault of the lousy mental model in, not the computer.

Is the model valid?

I think we’ve established that strategists always use models, whether mental or computer-based. Sooner or later someone asks how you know your model is valid. I think that mostly they want to know about numerical validity and accuracy; does the model work, has it been tested. That’s a good question we’ve all been trained to ask. We’ll talk about that next. However, I think it’s even more important to ask about conceptual validity; in effect, should the model work, does it make sense. We’ll talk about that in the part about rephrasing the question, just a few terrific paragraphs from now.

Actually, people ask the validity question only with computer-based models. Mental models are invisible, don’t feel like models, and are self-evidently correct to their owners. People may contest other people’s conclusions, but, as we said a while back, people rarely ask why they reach different conclusions from the same data.

When people ask if a model is valid they usually mean to ask whether it fits known data. The validity question is fair, but fitting known data is often not a useful test.

It’s always possible to put together a model that fits known data. If you expect the future to look just like the past, then such a model might even be useful. (Assuming it didn’t violate good model-building hygiene, such as using up too many degrees of freedom. See also Predictable Competitors.) But it’s not especially interesting to model a future that will look just like the past. The time you really need a good model is when the future will not look like the past. Unfortunately, there are no data about the future, so the does-it-fit-the-data test isn’t available. (See also More Internet Users than People.)

I recommend rephrasing  the question to shift from numerical validity to conceptual validity. Rather than ask if a model is accurate, ask if it is sensible. Ask how it works, ask what it takes into account, ask on what paradigm it rests. Remember that making strategy decisions is not about accounting, trend lines, or forecasting. It is about strategy, and the models you select should work with strategy concepts. (See also Pundits and Stress and With All This Intelligence, Why Don’t We Have Better Strategies?)

The bottom line

The ultimate point about validating models — and about selecting models and especially about using models — is this. You are going to make decisions no matter what. The relevant question is whether you can make a better decision with the model than without it.

No model is perfect and there are no guarantees of fabulous future performance. Improving the odds of success is all you can hope to get by using models. Fortunately, it is a lot to get. It’s the difference between the gambler and the casino.

This essay talks about models themselves. The next in the series, What The Model Says, discusses which models are worth listening to. The final essay, The Model Whisperer, connects mental and computer-based models to techniques for developing competitive strategy.

This brief essay highlights key themes from my Numbers Gone Wild workshop at the 2010 SCIP International Annual Conference on March 11, 2010. It does not replay the workshop, as the workshop was energetically interactive, verging at times on raucous; well, as raucous as competitive-intelligence professionals get. Not to mention that the group enjoyed numerous punch lines and shockers, and I wouldn’t want to spoil the delights and surprises if someday you experience the workshop yourself. Rather, this essay focuses on concepts and conclusions, which will shock and delight better than would a play-by-play recap.

Introduction: Crazy

Having numbers drives us crazy.

Lacking numbers drives us crazy.

Seeking precision drives us crazy.

Living with imprecision drives us crazy.

In short, we are going crazy.

Numbers surround us. Even people who prefer qualitative methods (“quals”) use numbers every day.

Numbers bug us. Even people who like quantitative methods (“quants”) criticize numbers every day.

In this workshop prudent people explored the unintended ways in which numbers going wild cause us to suffer. We suffer as a result of the bad decisions, and the collateral career damage, that can come from untamed numbers.

How to Like Numbers

I like most numbers. When I say “most” I don’t mean that I get along with all the digits except 7 and 8. I mean that numbers are literally the only way that we can gain certain insights, knowledge, and even inspiration, and that makes them likable.

Of course, the presence of numbers does not guarantee insight, knowledge, or inspiration. On the other hand, the absence of numbers limits us to anecdotes, impressions, and hypotheses. So, we want numbers.

I like most numbers. I don’t like most factoids. Here’s an example: “GM’s market share went down at the same time the reliability of its cars went up.” That statement is true and over many years it was measured precisely in cars sold and defects per hundred vehicles, but it is not likable. Why not? Here’s one of several reasons: because it is context-free. Context would tell us what else was happening while GM improved reliability and lost market share. Context would reveal that many more-reliable Japanese cars had entered the market and the other American producers were improving reliability too. GM almost certainly would have lost more market share if it hadn’t improved reliability.

Our unlikable factoid illustrates how we can decide which numbers to like and which not. The relevant criterion for likability is not whether a number is “actionable.” After all, the factoid is actionable: advise GM management to regain market share by making their cars less reliable. The relevant criterion is whether a number is sensible.

“Sensible” needs a big tent. It includes context, robustness, and quality of analysis. It does not imply or require approval. You might not like a number, but numbers are not popularity contests. (Which is amusingly ironic, if one is easily amused, because popularity contests are judged with numbers.)

The big tent of sensibility also does not imply or require precision. It doesn’t matter a whole lot whether GM’s market share declined 11.2% or 11.3%. No one would breathe a sigh of relief to find the loss was the former and not the latter. Moreover, precision is not even an option for many numbers, such as all numbers about the future.

Notice that the sensibility criterion separates the wildness of a number from what we do with that number (e.g., approve of it or take action on it). That is a good thing. When we humans get wrapped up in strategy debates and decision-making, we confound the goodness of a number (its sensibility) with our approval of the number (its value in getting our idea adopted). We slip into judging the goodness of a number by the support it lends to our idea, rather than judging the goodness of our idea by the support it gains from a number. It is to avoid confounding sensibility and approval that I design the structure of my strategy-simulation models before calibrating the models with data, and that I calibrate the models before simulating strategies with them.

Models in Our Heads!

Think about your business. What would happen to its profits if its prices were to rise 5%?

To answer that question you will execute a series of unconscious cerebral calculations to forecast the net results of many interconnected moving parts. How do I know you will do that? Because you will have an answer to my what-would-happen question, meaning you will promptly predict how prices will prod profits.

Let’s think about that. To promptly predict how prices will prod profits you will assess:

• Competitors’ reactions to your price increase. Will they follow you up or will they tout their new price advantage? How quickly will they move, how far will they go?
• Customers’ reactions to the price increase. What will happen to your business’ sales in the short term? In the long term? Will customers expect a better product for the higher price? Can you market the increase in a soothing way? Will you try to hide the increase via smaller packages or hidden fees? Will customer loyalty dampen or conceal the effects?
• Given the changes (if any) in your business’ volume, what happens to costs? Are fixed costs fully covered? What are the implications for contracts with labor, distributors, suppliers, and shipping? What about sales support and customer service?
• And more.

It would be extraordinarily difficult for you to take all of those factors into account in your head, not to mention doing the arithmetic. And yet you will proffer a prompt prediction to the what-would-happen question. As would most business professionals. You and I hear it all the time; just listen to debates about what works and what to do.

That prediction comes from what’s called a mental model. It’s just like a computer model, except it happens in one’s head and it operates with a lower degree of computational accuracy.

A lower degree of computational accuracy? Yes. Try this. You sell a product for $129 per unit. You sold 51,500 last quarter. The market is growing at a compound annual rate of 3.8%. What will be your revenue next quarter? Write down your answer. That, by the way, is a much simpler question than the one I posed before, the one about raising prices. Even so, you probably had a confident opinion about the first question while feeling helpless to answer the second without electronic assistance.

Mental models are always switched on, even within the quals wandering among us. They matter because we use them to predict the results of actions we might take and to judge the validity of the computer models we might use. We can and should take these steps to use both mental models and computer models without producing garbage-out:

• Mental models are generally unconscious. Make them explicit. Draw them, discuss them, think through the assumptions and principles that drive them. Question mental models and their owners, respectfully, just as we question computer models. Separate their conceptual sensibility from the numbers they produce, and focus on the former first.
• Because humans tell computer models how to think, computer models think like humans. If you believe a financial analysis is appropriate, you use a financial model in which the computer has been taught that profit equals sales minus costs. There’s a flip side: when you use a financial model, you implicitly aver that the problem at hand is properly addressed by financial analysis. Worries about “garbage in” apply to the choice of analysis at least as much as to the numbers fed in.

Important point: both mental models and computer models are about thinking. Both reflect how we believe the world works.

By the way, the answer to the arithmetic-accuracy question is $6,705,733.32.

Oh, Them

Let’s compare models. We’ll go back to the $129 product arithmetic and contrast it with the raise-prices prediction. There’s a critical difference between the two.

• Yes, the $129 product arithmetic is more precise. We have actual numbers. But although it’s a real difference, it’s not a critical difference.
• The $129 product arithmetic can be calculated, monitored, and verified. Another real difference, though not yet critical.
• The $129 product arithmetic implicitly and invisibly implied that the solution was a matter of financial analysis. I reinforced that perception by 1) supplying an answer 2) that was calculated to the penny. The raise-prices prediction implied financial analysis much less, and perhaps not at all. That’s critical.

Review the list of assessments you’d make in the raise-prices prediction. You might assess well and you might assess badly, but the way you would think about that problem would probably differ dramatically from the way you would think about the $129 product arithmetic.

The superficial precision of the $129 product arithmetic nudged you toward a method of analysis. (That’s known as “framing” the problem.) Thus, precision in could cause garbage out. Why garbage? Because the $129 product arithmetic should raise exactly the same issues about competitors and customers as the raise-prices prediction, and using a financial-analysis approach would completely, presumably unintentionally, and potentially disastrously ignore competitors and customers. Oh, them.

Conclusion: Not Crazy

The cure to our initial craziness is not to dispense with numbers wholesale, nor is it to embrace them unconditionally. The way to prevent garbage-out is not to scorn precision, nor is it to demand it. The answer is not either-or or some mythical “sweet spot.”

The answer is to deploy the best of both worlds, thinking strategically and calculating rigorously. Not perfectly or exhaustively, but consciously and vigorously.

That means being careful as we frame problems and choose models. It means remembering that there’s always a model, whether it’s in your head or in your computer. It means taking relevant factors into account, even imprecisely, because omitting relevant factors hurts more than using them imprecisely.

I know it’s possible because I’ve done it in the strategy models I’ve built in my career, and I’m not the only one who’s done it.

Let’s close with lessons from the remarkable Deep Blue.

Garry Kasparov, the world’s reigning chess champion at the time, was defeated in 1997 by IBM’s Deep Blue supercomputer. There are several points we can make about that match.

First, Deep Blue was not programmed with trend lines, gap analysis, or financial fundamentals. Of course not; it was a chess match. So why use trend lines, gap analysis, or financial fundamentals to analyze competitive-strategy problems?

Second, Deep Blue was a machine but it thought like a human. That’s because humans programmed it, with their chess-playing knowledge and expertise. Computers do not contest us; they express us. They are amplifiers for our thinking the way power tools are amplifiers for our muscles.

Third, it’s virtually certain that none of the individuals who programmed Deep Blue could have beaten Kasparov. Their combined talent, though, did. It’s also virtually certain that Deep Blue would have beaten any single person whose knowledge and expertise it contained.

Finally, we know that Deep Blue didn’t win every game in the match, nor (obviously) did Kasparov. But there’s a question deeper than which of them was best. The deeper question is this: who or what could beat a Kasparov/Deep Blue team? The value of numbers and models is not numbers and models. The value is better business decisions that help you succeed.

How to avoid garbage-out on your way to better decisions? Think about it.

Workshop
ACS’ Mark Chussil will deliver a workshop, “Numbers Gone Wild: Or, Precision In, Garbage Out,” at the SCIP 2010 Conference in Washington, DC. See below for an overview of the workshop.

Update: See Numbers Gone Wild (the essay), which shares some themes from the workshop, delivered on March 11, 2010.

Webinars
ACS is planning its upcoming schedule of webinars about business war games, strategic thinking, and other topics of interest to strategists. If you would like to be notified as those webinars become available, please let us know.

The Numbers Gone Wild workshop
I’m just an everyday guy who runs 20,000,000 simulations before breakfast. Doesn’t make me break a sweat, though my computer feels a bit warm. I like numbers and I like learning the things that only numbers can reveal or teach.

That said, numbers also drive me crazy. We’re surrounded by pointless numbers, pseudo-precise numbers, even silly numbers, and as a result we make pointless decisions, pseudo-precise decisions, even silly decisions.

The answer isn’t the quant’s digital paradise. That’s so even though my mini surveys show people believe the best way to get better decisions is to get more-precise data. The answer isn’t the qual’s number-free zone. That’s so even though some of my best friends are innumerate. Rather, the answer is in how we think about numbers and in the numbers we choose to think about.

No statistical expertise is required for this workshop. Quals and quants will both be comfortable and entertained.

This workshop uses a series of interactive exercises and games to demonstrate how our misuse of numbers leads to strategy mistakes. We’ll talk about those mistakes in the context of mental models, precision, spreadsheets, gap analysis, trend lines, paper-folding, groupthink, survivor bias, analyzing novel situations, and the Strategist’s Dilemma. We’ll talk about the mistakes incumbents make that let upstarts win. And no, the mistakes we’ll talk about are probably not those you’re expecting. For example, although garbage in, garbage out is a problem for spreadsheets, it’s almost trivial as these problems go.

This is not a workshop about calculating, decimal points, or the difference between correlation and causation. It is a workshop about thinking, in particular strategic thinking. It is a workshop about getting a fresh view on common challenges. And ultimately it is a workshop about making much better strategy decisions.

See also Precision In, Garbage Out

In this essay we will analyze a difficult problem that’s in the news. It’s about how our beliefs and assumptions guide our analysis of a knotty problem. It’s relevant to anyone who works with data, even if — or perhaps especially if — answers seem obvious.

My home town, New Haven, Connecticut, has been in the news. Not because it’s my home town. Rather, because of a reverse discrimination lawsuit (Ricci v. DeStefano) brought by firefighters against the city. The case is in the news also because Judge Sonia Sotomayor, who’s been nominated for the U.S. Supreme Court, joined her colleagues on the Second Circuit Court of Appeals in a unanimous decision backing the city.

Ricci v. DeStefano is now before the Supreme Court. (Update: for their decision, please see the end of this essay.) I thought it might be nice to bring some cold, rigorous thinking to the hot, emotional case.

The case concerns tests used to promote firefighters to lieutenant or captain. Although the legal arguments focus on which parties have which rights and obligations, the core question (which we might hope is relevant) is whether the tests actually were discriminatory.

Here are the numbers:

• 41 people passed the captain’s exam: 25 white, 8 black, and 8 Hispanic. The city would have to promote the 9 people with the top scores. 7 were white, 2 were Hispanic, none were black.
• 77 people passed the lieutenant’s exam: 43 white, 19 black, and 15 Hispanic. The city would have to promote the 10 people with the top scores. All were white.

The city contends that the apparently too-high percentage of whites being promoted by their test scores is evidence of racial discrimination, and so they threw out the test results. The firefighters say that throwing out the test results reverse-discriminates against those who scored well on a fair test.

I decided to calculate whether the high percentage of white promotions was statistically “too” high. If it is statistically unlikely that whites would do so well and non-whites not so well, we’d have evidence that the tests might have been discriminatory (the city’s position). If the odds are high that it the results could happen by chance or merit, we’d have evidence that the tests were not discriminatory (the firefighters’ position).

Note that analysis can say nothing about the intentions behind the tests. Note also that analysis cannot prove that the tests were discriminatory (or not) in some absolute-truth sense. Still, if the odds strongly favor one side or the other, that should count for something. A reasonable person would and should draw different conclusions about X if the odds of X are 1% and if the odds of X are 99%.
 
I wrote a computer program that looked at every possible way to distribute 41 people in 9 slots (the captain’s exam) and every possible way to distribute 77 people in 10 slots (the lieutenant’s exam). Then, it looked at how many of those possible ways matched the actual racial distribution of the results.
 
There are 350,343,565 possible combinations of 9 winners on the captain’s exam. Of them, 13,459,600, or 3.8%, had 7 whites, 0 blacks, and 2 Hispanics. Another way to look at the results is how many had 7 whites and 2 non-whites. Under that test, 57,684,000 combinations match 7 whites and 2 non-whites, or 16.5%.

In statistical analysis, 5% is a common threshold for “significance;” that is, 1-in-20 odds, a fairly reliable result. (More-stringent analysis uses 1%.) Thus, the 3.8% supports the city’s case, and the 16.5% does not.

Then there’s the lieutenant’s test. There are (this is not an exaggeration or a joke) 1,096,993,404,430 possible combinations of 10 winners out of 77 people. Of that trillion-plus, 1,917,334,783 fit the 10 whites, 0 non-whites outcome. The odds of that are far below 1%; to be exact, the odds are 0.17%. That suggests it was not an accident that 10 whites got the 10 top scores. Why it happened — the test itself, the scoring, self-selection among those who took the test, something else — is a different question, about which neither the analysis nor I make any statement. (I didn’t mention merit as a reason why it happened. We’ll come back to that.) All we can say is that there’s only 1 chance in almost 600 that such an outcome would occur by chance. That’s like guessing a coin toss correctly 9 times in a row: it can happen but you wouldn’t bet on it. Those results support the city’s case strongly.

That said, 1-in-600 odds don’t prove there is discrimination. Those test results could happen by chance, especially if multiple cities use the same test or the same city uses the test multiple times. It’s like winning the lottery if you play enough times or dying on an airplane if you fly enough times. And again, there’s the question of merit.

It gets more complicated. We know that New Haven took pains to create a test that wouldn’t racially discriminate. Assuming that they were sincere and at least partially effective in their efforts, that should raise our confidence that the test results happened by chance (the firefighter’s position), not by discrimination (the city’s). How much should we raise our confidence that the test was not discriminatory? I don’t know. A place to start might be to compare the results of the contested exam with the results of previous tests, or to look at other cities’ test results.

Here’s a different complication: how do we define or discern discrimination? Presumably it would show up as an unfair boost, not unlike steroids, rather than as a blatant gift. Let’s try an experiment. What if, for instance, whites were surreptitiously given slightly higher scores on the tests than blacks or Hispanics? I don’t know how that would be done, but let’s assume that there was a clever way. The average score on the lieutenant’s test for whites was 71.8, for blacks 63.8, for Hispanics 63.6. An unfair boost is one of several possible explanations for that difference. The existence of the difference does not prove the difference was unfair or even statistically reliable, though it begs to be studied more. Regardless, what if we split the difference on the averages and subtract 4 points from every white candidate’s score? How many of the top 10 scorers would be white?

Answer: still 10.

Subtracting 4 points from the whites’ scores is arbitrary, and it could be argued that it introduces clear bias in an attempt to eliminate assumed bias. Even so, 4 points doesn’t change the promotion list. That’s an argument in favor of the firefighters who brought the lawsuit, though it doesn’t prove much.

What if we completely erase the average differences among the groups by subtracting 8 points from the whites’ scores? We’d have 6 white winners and 4 black winners, an argument in favor of the city. And we still haven’t proved much.

Neither the 4-point experiment nor the 8-point experiment proves anything about the presence or absence of discrimination. They merely show the sensitivity of the promotions to presumed systematic bias of a certain number of points. The experiment is about the size of the arbitrary subtraction, not about discrimination. The experiment comes down to whether the experimenter believes that 4 points, or 8 points, or 0 points, or 2.736 points, or 12.345 points, is the right adjustment for the differences in average scores. It’s an analysis based on an assumption. (If there are data that support a real adjustment, that’d be another matter entirely. I don’t know if any such data exist.)

Let’s try another approach. Forty-three whites passed the lieutenant’s exam, along with 19 blacks and 15 Hispanics. Given the much larger number of whites, we’d expect that there would be more variation among them than within either of the other two groups. That happened: there was a wider range of scores among whites. Some of those scores were at the top end. That supports the firefighters’ case.

At last, here’s the merit issue I’ve been promising you. Calculating the 1-in-600 odds started with the implicit and more-or-less invisible assumption that all the people who passed the test got the same score. It’s a direct consequence of treating each of the trillion-plus combinations of winners as equally probable. (Did you spot that assumption? I didn’t until I got pretty deep into my analysis.) In effect, my calculations answered the question “how probable is it that equally qualified people of different races would produce 10 white winners (the lieutenant’s test) or 7 white and 2 Hispanic winners (the captain’s test)?” But how do we know people’s qualifications? That’s what the tests are supposed to reveal. And if the tests reveal merit, then the test results would be right, by definition. But I don’t know if they do (perhaps someone else does); some tests work and some tests don’t. For now, we can only make assumptions.

Presumably the New Haven Fire Department believes the tests measure something of value. On the other hand, presumably no one believes the tests are perfect. So what should we conclude?

Alas, our analysis is not conclusive. If we assume that the candidates were equally qualified, more or less, the 1-in-600 calculation is pretty compelling in favor of the city. Ditto if we assume the tests and scoring were slanted, intentionally or not, toward the white candidates. On the other hand, the wider variation in the larger group, the less-than-overwhelming 1-in-26 odds on the captain’s test, and the city’s previous efforts to ensure fair tests argue in favor of the suing firefighters.

Most important, there’s the question of whether the tests measure merit. If they do, the firefighters’ case is strong. If they don’t, the city’s case is strong.

The bottom line: no definitive answer yet.

Let’s put New Haven aside and up-level the discussion.

When I ran my computer program, I was sure of my conclusion: New Haven is right, the firefighters are wrong. As I wrote this essay, though, I thought about questions my dear readers might ask, and those questions made me think. I questioned my methods, assumptions, and conclusions. I went back and forth as I crunched numbers every way I could imagine short of making this essay my career. So much for cold, rigorous thinking; instead, I got a cold, hard dose of unwanted humility as my conclusion morphed into I don’t know. More data might help, especially information about the validity of the tests. I don’t have those data, and even if I did, there’s only so much time for analysis before we must make decisions. That’s true in government and in business.

That I have not come to a clear conclusion about Ricci v. DeStefano doesn’t mean this exercise (and your faithful reading) has been useless. Quite the contrary; we can come to conclusions about how we come to conclusions. This exercise has:

• Made me less likely to jump to a tempting conclusion based on a short stack of factoids.
• Reminded me that numbers don’t speak for themselves (1-in-600 odds), that analysis reflects basic assumptions (equally probable combinations of winners), and that statistical processes are always at work (wider variations in larger groups).
• Taught me to actively, deliberately look for contrary data and ideas and to keep asking how my analysis could be wrong or incomplete.
• Helped me formulate different ways to solve analytic problems. For example, when there is no clear conclusion, ask a different question. “Which mistake would I rather make” is a good one. “What would a smart reader say” is another.
• Shown me that even though “proof” may be an unattainable standard, we can avoid simplistic answers.
• Proven to me that inconclusive data can lead to more-thorough thinking.

Both sides care about the truth, and we have learned that the data we’ve seen so far are not sufficient to tell us the truth. We have learned that we can debate the promotion patterns and test scores as long as our voices hold out but those data alone do not reveal the truth. We have learned that making a decision relying solely on the data we’ve seen so far will be at least partially the triumph of persuasion or ideology. Finally (and this is both important and exciting), we have learned what else we need — data on the validity of the tests themselves – to make a decision using reason and analysis.

Update. On June 29, 2009, the Supreme Court ruled in favor of the firefighters in a 5-4 decision written by Justice Anthony Kennedy. The Wall Street Journal reported that Justice Kennedy said employers “must show a ‘strong basis in evidence’ before ignoring results of employment-related tests.” As we saw in this essay, the evidence was mixed, and whether discrimination had been present or absent, it would have been difficult to prove or disprove.

Update. August 5, 2010: Judge Halts New York City From Using Exam Results to Hire Firefighters. Here is the district court’s full decision. Of particular interest are the tight “bunching” of test scores near the top of the range and the use of rank ordering to select candidates to hire. In other words, is a score of 98 actually better than a score of 97.

Further reading
Steven D. Levitt and Stephen J. Dubner, Freakonomics.
Leonard Mlodinow, The Drunkard’s Walk.
John Allen Paulos, Innumeracy.
Jay Russo and Paul Schoemaker, Decision Traps.
Nassim Nicholas Taleb, Fooled by Randomness.
See also Marvelous Techniques.

We ran ACS’ Decision Tournament™ for pricing in conjunction with SCIP Oregon. You can see a description of the tournament along with interesting results and unique charts.

To date, 156 strategists have participated in the pricing tournament, leading to over 5,600,000 simulations (you’re right, it’s not an Excel spreadsheet) and our shocking blog post When I Was Wrong. If you would like to experience the simulation too (and, afterward, get a white paper designed for simulation participants), please write to info@whatifyourstrategy.com. It’s free and fun.

(Update. As of early 2010, nearly 300 strategists have participated in the tournament.)

All 156 participants in the tournament wanted to win and thought their strategies would win. How do we know? If they didn’t think their strategies would win, they would have chosen different strategies. Nonetheless, the outcomes of their strategies varied hugely. Testing their strategies through strategy simulation — in effect, a massive business war game conducted inside a computer — helps identify which strategies do well and which do badly.

In these times of economic crisis, with the stakes so high on pricing and other decisions, there’s more need than ever before to act creatively, and to stress-test action options before risking time, money, and careers on them.

We thank Sean Campbell and Scott Swigart of Cascade Insights, who chaired and organized the SCIP Oregon meeting, for inviting ACS to conduct the simulation.