We will end this essay with a valuable point about costs and prices. We will begin with an enigma about assembling products at home.

As I home-improve my home, I come across products whose directions warn me not to “overtighten” something. This confuses me. How am I supposed to know how much tightening is too much? If something shatters, cracks, or crumbles, I figure I’ve overtightened it, but that information is tardy and not helpful. Remember, I don’t want to undertighten it either. I don’t want it to fall apart and spill a Caution! Awfully Hot! liquid on me.

A pleasant alternative would be “go ahead and tighten until you reach the limits of your strength.” To do that, the item being tightened would have to be strong enough to withstand my formidable strength, and we all know that sturdy is out in these cost-conscious times (which apparently have been in effect for 30 years or so, based on how often we are warned of the delicacy of items about to be tightened).

I think the elegant beauty of “do not overtighten” can only be understood from the perspective of a lawyer. (Your Honors, I hereby state for the record that I am not, have never been, and expect never to be against lawyers. I almost became one myself. Ipso facto. Habeas corpus. De minimis non curat praetor.) Or understood from the perspective of folks in product support, who in turn are dealing with a situation created by cost accountants (about whom I also have nothing against). Let’s replay the scenario.

Customer: “I was putting the product together and it broke.”
Support: “When did it break?”
C: “Sunday.”
S: “No, at what point in the assembly process? Were you tightening something?”
C: “Yes.”
S: “You must have overtightened it. The instructions specifically state not to overtighten it. Bad customer. There is no warranty, express or implied, on overtightened items. We would be happy to sell you a replacement. Want my advice?”
C: “Yes, please.”
S: “Do not overtighten it.”

Businesses tend to overtighten. They do it because they’re led there by simple, persuasive logic, which we can boil down to this: it is cheaper to print “do not overtighten” on the instructions than it is to supply products that can withstand formidable strength. It is even virtuous to do so, because driving excess costs out permits lower prices, which makes customers happy. That’s true. Until we overtighten. Unfortunately, the dividing line between nice-and-tight and overtight is as hard to discern in business as it is in home improvement.

Overtightening in business leads to being unable to get an airline seat for days if your flight is canceled. (See “Police Called to Quell Unruly Passengers at J.F.K.” Notice that the airline blames the weather, not overtightening.) Overtightening leads to being put on hold for hours. Overtightening leads to being vulnerable to hiccups anywhere in a long supply chain. Overtightening leads to nutritional quality gradually being subtracted from food.

Nobody intentionally overtightens. Probably nobody thinks he or she does overtighten. We’re just listening to that eloquent cost analysis showing how much money we lose when there’s an empty seat on an airplane, when phone operators are sitting idle, when we don’t outsource to Mars, and when we compare natural proteins to polyglutamatinousbiphosphorescentcosamine.

You will notice that this essay is not entitled “In Defense of Wasteful Practices.” I am not defending waste; I am in favor of efficiency and productivity. All I’m saying is that cost analysis makes an implicit assumption: if saving $1 is good, then saving $2 is twice as good. It makes another implicit assumption: saving the next dollar is equally good as saving the previous dollar. In other words, the tools we use to gauge the goodness of tightening make it hard to tell the difference between nice-and-tight and overtight.

Some companies, notable for their rarity, explicitly reject overtightness. I had a Sears Craftsman ratchet wrench that I broke, perhaps by overtightening some poor thing. Sears offers a lifetime warranty on Craftsman tools. I took the wrench to Sears, found my way to the tool department, and presented the wrench to a friendly Searsperson. I said that I’d bought it a long time ago, far, far away, and it broke. Before the word “broke” had faded from my lips, I had a replacement wrench in my hand, along with a receipt documenting that there was no charge. Similarly, there’s Nordstrom, and LL Bean, and Chelsea Audio Video, which is even more guy-fun than the tools department at Sears.

So how can you avoid that sickening feeling that comes right after you’ve overtightened?

• Beware of the death of 1,000 tightenings. I believe it was John Allen Paulos, in one of his marvelous books about innumeracy, who gave a terrific example of how we devolve from nice-and-tight into overtight. (If it wasn’t Prof. Paulos, please correct me.) Imagine that customer surveys show product A is statistically indistinguishable from product B, and product B is indistinguishable from product C, and C from D, and D from E. It does not follow that A is indistinguishable from E. If A through E involve successive minor cuts in quality, it may appear that customers won’t notice or care about each cut, but when you add them all up A is far from E. Think about airlines: an inch of legroom here, a baggage fee there, no more meal service, and pretty soon you just want anesthetics. (Available for your convenience for a small extra charge.)
• Don’t filter out what you need to see. “Customers aren’t asking for something else”? How do you know? Multiple-choice surveys make it hard for oddball ideas to get through. People with complaints get shuttled to the complaint department, which emphasizes placating; how do the complaints get to R&D? Comparisons with competitors — “we perform in accordance with accepted industry standards” — are meaningless if you and your competitors behave the same way. Which gives creative upstarts the opportunity to beat coasting incumbents. (See Further Reading, below.)
• Why, oh why? We all make assumptions; the trick is periodically to ask why oh why they work for your business. Here are some examples: We must sell out our capacity; it’s better to sell a perishable good at a low price than to have it go unsold; we should scale capacity to fit peak demand; we should maximize demand; customers are more sensitive to price than quality. I’m not saying those assumptions, and others, are right or wrong. I am saying they are assumptions (possibly supported by data; see Further Reading), and before we continue to risk our businesses on them we should ask why we continue to believe them. I’ve found in business war games I’ve conducted and strategy simulations I’ve built that a good test question is this: will our strategy work if our competitors do the same thing we plan to do?

Competitive strategy. Some assembly required. Do not overtighten.

Further Reading
About overtightening: Gross Galactic Product.

About upstarts versus incumbents: With All This Intelligence, Why Don’t We Have Better Strategies?

About assumptions: House, MBA, More Internet Users than People, and Doesn’t Make Sense.

About being misled by well-intentioned data: It’s Working! and Self-Fulfillment

About the strategy equivalent of the Tower of Babel: Motor Swilling Forbidden.

Workshop
ACS’ Mark Chussil will deliver a workshop, “Numbers Gone Wild: Or, Precision In, Garbage Out,” at the SCIP 2010 Conference in Washington, DC. See below for an overview of the workshop.

Webinars
ACS is planning its upcoming schedule of webinars about business war games, strategic thinking, and other topics of interest to strategists. If you would like to be notified as those webinars become available, please let us know.

The Numbers Gone Wild workshop
I’m just an everyday guy who runs 20,000,000 simulations before breakfast. Doesn’t make me break a sweat, though my computer feels a bit warm. I like numbers and I like learning the things that only numbers can reveal or teach.

That said, numbers also drive me crazy. We’re surrounded by pointless numbers, pseudo-precise numbers, even silly numbers, and as a result we make pointless decisions, pseudo-precise decisions, even silly decisions.

The answer isn’t the quant’s digital paradise. That’s so even though my mini surveys show people believe the best way to get better decisions is to get more-precise data. The answer isn’t the qual’s number-free zone. That’s so even though some of my best friends are innumerate. Rather, the answer is in how we think about numbers and in the numbers we choose to think about.

No statistical expertise is required for this workshop. Quals and quants will both be comfortable and entertained.

This workshop uses a series of interactive exercises and games to demonstrate how our misuse of numbers leads to strategy mistakes. We’ll talk about those mistakes in the context of mental models, precision, spreadsheets, gap analysis, trend lines, paper-folding, groupthink, survivor bias, analyzing novel situations, and the Strategist’s Dilemma. We’ll talk about the mistakes incumbents make that let upstarts win. And no, the mistakes we’ll talk about are probably not those you’re expecting. For example, although garbage in, garbage out is a problem for spreadsheets, it’s almost trivial as these problems go.

This is not a workshop about calculating, decimal points, or the difference between correlation and causation. It is a workshop about thinking, in particular strategic thinking. It is a workshop about getting a fresh view on common challenges. And ultimately it is a workshop about making much better strategy decisions.

See also Precision In, Garbage Out

This is my modest contribution to a vigorous LinkedIn discussion about Twitter. I edited it slightly to make me sound smarter than I originally did. Some in the discussion argued good-humoredly that Twitter is an important new force. Others argued good-humoredly that it is a useless new farce. Both groups had copious supporting anecdotes and analogies.

As you read here, think of the arguments about competitive-strategy options you’ve heard.

Fascinating. This is fun. Thank you all. Seriously, thank you.

This is just like politics: anecdote and analogy. And, just like politics, there’s always an anecdote and analogy that supports one’s side. I happen to find one side’s anecdotes and analogies more compelling than the other side’s, but who cares? I haven’t any followers. [If you “tweet” well, you accumulate people who “follow” your every 140 characters.] I don’t tweet, therefore I am not.

Something always works, or at least seems to work, and the people who jump on its bandwagon appear prescient. Something always fails, or at least seems to fail, and the people who stay off its bandwagon appear prudent. This is why we have, and need, the scientific method.

Okay, okay, the quick movers with itchy texting fingers are groaning at the fuddy-duddy who wants actual evidence of efficacy. The lack of such evidence is why I don’t buy into, say, astrology and homeopathy. I do, however, appreciate my late grandfather’s advice: if you eat borscht for 90 years, you’ll live to be old. He missed by six months. He should have eaten borscht longer, I suppose.

But back to Twitter. I appreciate that evidence of efficacy may be hard to come by, especially if no one is collecting evidence other than the bandwidth consumed or the number of numb-fingered twitterers. I would provisionally settle for a testable causal theory that, if true, would explain why Twitter would work under X conditions and not work under Y conditions. We also have to define what it means to “work.” Boost sales? Improve the quality of decisions? Make our lives happier?

In lieu of evidence and a causal theory, I say that if you like to tweet, go ahead and tweet. You don’t need to justify it — and you cannot justify it —any more than you need to justify a preference for cabernet sauvignon over pinot noir. Recognize that the burden of proof (not the burden of anecdote) is on you if you want to convince the rest of us that we “should” tweet.

If you don’t like to tweet, don’t tweet. You don’t need to justify it either, nor can you.

Besides, who needs to be in such constant communication? But enough of that. If you’ll excuse me, I simply must check my email.

A poetic note about the lack of evidence. “I hope that while so many people are out smelling the flowers, someone is taking the time to plant some.” — Herbert Rappaport (1913-1999?).

Further reading: House, MBA and “Brain Food.”

This post is an invitation for you to attend a free webinar presented by ACS’ Mark Chussil on behalf of the LinkedIn group “Corporate Planning & Global Industry Segmentation.”

The webinar will be on Tuesday, November 17, 2009, at 1:30 pm east-coast time, 10:30 am west-coast time. It will be one hour in duration. You can access video through a web link that you will receive 30 minutes before the webinar begins. (See “How to sign up,” below.) You can access audio by dialing a phone link that you’ll receive with the web link.

The topic is Winning: The How and Why of Business War Games.

Overview

Business war games sound cool. That’s because they are. Business war games are more than a way to escape the tedium of gigabyte spreadsheets and death by PowerPoint. They are a way to put your knowledge of your business to better use. With business war games strategists can achieve surprising new insights into businesses they know well. And by the way, business war games are not about war.

Agenda for our interactive presentation

• Why business war games work where conventional strategy development doesn’t.
• What business war games are good for.
• The essential elements of successful business war games.
• How to design or select a business war game for your company.
• What strategists have learned from business war games; in other words, war stories.
• Q&A

About the presenter

Mark Chussil is Founder and CEO of Advanced Competitive Strategies, Inc. and a pioneer in the field of business war-gaming and strategy simulation. He is also a Founder of Crisis Simulations International, LLC. Mark’s technologies have won a patent (for CSI’s DXMA™ crisis simulator) and an industry award (for ACS’ ValueWar™ strategy simulator). A highly rated, thought-provoking, and entertaining speaker, Mark has lectured and conducted hundreds of business war games on six continents. He has written two books, chapters for five others, and dozens of articles and essays. He earned his MBA at Harvard and his BA at Yale.

How to sign up

You can sign up in either of two ways: through ACS or directly through the LinkedIn group. Whichever you select, please sign up before November 17.

Signing up through ACS

Send an email to info@whatifyourstrategy.com saying that you want to participate in the webinar. ACS will inform the group moderator and email access information to you 30 minutes before the presentation begins.

Signing up directly through the LinkedIn group

• Register on LinkedIn if you don’t already have a profile.
• “Search Groups” for “Corporate Planning & Global Industry Segmentation.”
• Ask to join that group by clicking on “Join this group,” displayed at the right of the group description.
• After the group’s moderator sees and approves your request (usually the same day), visit the group and find Discussion “Special Presentation: Tuesday, Nov 17th.”
• Add a comment to say you will attend.
• On November 17 you will receive an email 30 minutes before the presentation with the call-in number and the URL for the webinar.

Please feel free to share this invitation with your colleagues.

We hope you’ll join the webinar!

In today’s Wall Street Journal (October 16, 2009), we find these contrary positions in “Safeway Shifts Tactics in Grocery Price War:”

“Last month, [Safeway CEO Steve] Burd conceded had the chain moved quicker to lower prices, it would be ‘doing a bit better than we are now.’”

“Increased use of promotions ‘destroyed our gross margin,’ Supervalu CEO Craig Heckert said last month at a Goldman Sachs conference. ‘More items really cheap don’t bring in more people.’”

There is truth in the notion that focus is valuable and in the notion that diversification is valuable. There is truth in leadership and in delegation. There is truth in caution and in action. There is truth in mass marketing and in segmentation. There is truth in price and in quality. Pick a business model, pick a buzzword, pick a fad; there’s probably truth in it, and in its opposite.

There’s also falsehood, or at least incompleteness, in each of those perspectives. Both Burd and Heckert expressed dissatisfaction with their results. Each CEO wished, if just a little, that he’d done what the other had done… and what the other was blaming for his dissatisfaction.

Burd and Heckert are experienced, high-powered professionals with plenty of data, analysis, and advice at their disposal, yet they come to opposite conclusions. I’m not criticizing them; quite the contrary, I’m sympathizing with them because they’re trying to cure sick systems with many moving parts.

Let’s consult with Gregory House, fictitious M.D., of the eponymous TV show. If you like House, he’s the witty, incisive doctor whose cool, unrelenting pursuit of the truth cures the patient. If you don’t like House, he’s the nasty, sarcastic doctor whose cold, unrelenting pursuit of his ego cures the patient. Cool or cold, he cures. If I were sick enough to need House, I’d want House.

Yes, House (the show) is fictitious and, I’m told, unrealistic. Still, the show beautifully illustrates ways to get insight into tough questions rigorously and effectively.

(Here I’m going to talk about Burd/Safeway and Heckert/Supervalu as though I know what I’m talking about when, in fact, I don’t. All I know is what I read in that Journal article. Plus, of course, what I’ve learned in 30+ years studying competitive strategy. I’m just using those CEOs and companies to illustrate what we can learn from House.)

House (the doctor) begins by writing symptoms in modest-sized letters on a white board. That’s significant for three reasons: he can add, he can erase, and he keeps everything in view. He writes those symptoms without unnecessary detail or precision; he simply writes “high blood pressure” and “headache.” Partly that’s because watching someone scrawl numbers and details isn’t exciting television. Partly it’s because it keeps him and his team focused on the big picture.

Strategy equivalent: I’ve seen, and I’m sure you have too, strategists spend time debating whether to adjust a price by 1.0% or 1.5%. There’s time for that later. Let’s figure out first if, say, anyone is suffering from malignant prices.

For Burd and Heckert, the symptoms are apparently pain in the profit: sales below expectations and margins below expectations. (It is unclear from the Journal article whether we should say expectations, desires, or targets. We’ll just say expectations.) Prices are not symptoms; their role in the symptoms has not yet been determined.

House writes those symptoms without interpretations. He may write “high blood pressure” and “headache;” he doesn’t write “headache from high blood pressure.” He and his staff are free to link high blood pressure and headache as they search for a diagnosis, but they don’t presume it. Something else might cause both (a side effect from medication, a thrillingly obscure disease). A symptom might even be irrelevant or temporary; the headache may go away after the morning’s 17 espressos wear off. He also collects relevant information about the patient.

Strategy equivalent: relevant business, competitive, and market information, without interpretation. Relevant: we must, for example, understand our competitors because our performance depends in part on what they do. It’s not good enough to study only our business. Without interpretations: notice the difference in assumptions and implied action between “our prices are too high” and “our prices are 10% above competitors’.” Note too that SWOT analysis builds in interpretations. Is it a strength or a weakness to price 10% above competitors?

Burd and Heckert would presumably know that the other is suffering too. They’d know also that they have similar big-retailer business models. They can readily measure prices in their markets.

Then House does something remarkably powerful: he starts fights. He doesn’t hold back his critiques as his staff suggests possible diagnoses. His staff doesn’t hold back their critiques as he suggests possible diagnoses. The fighting gets heated (perhaps inevitable when the boss’s favorite expression is “you idiot”) but everyone is clear that the objective is to diagnose and cure the patient. There are a few rules that guide the fighting: 1) treat causes, not symptoms; 2) find causes that fit all the symptoms; and 3) the way you do your job is to be part of the fight.

Strategy equivalent: resist jumping to advocacy and conclusions, resist anecdotal reasoning. Brainstorming, scenario planning, and what-if simulations help a lot. Business war games are particularly terrific for conducting civilized and highly constructive fights.

Still, in business there is tremendous pressure to advocate rather than debate and to act rather than diagnose. Moreover, strategists simply do not have the business equivalent of a body of medical literature based on centuries of scientific investigation. Without that deep knowledge it becomes especially important to question assumptions and to look for causal mechanisms. What has to happen for a price cut to work for Supervalu and for steady prices to work for Safeway? Hint: it’s not just one or two items. Think it through. Do the assumptions and causal mechanism make sense to you?

Burd and Heckert implemented different price moves, presumably after tracing the effects of price changes on their financials, taking into account fixed costs and variable costs. Both are unhappy. That suggests the price moves might not be the cause, or at least not the sole cause, for the same reason that we’d say 17 espressos might not cause a headache if identical twins both got headaches but only one had 17 espressos. What else could it be? Perhaps they have something else in common that could causally explain their shared unhappiness. (If they don’t, then we’d treat them as separate cases; there are multiple causes of headaches.) Similar business models that are unsuitable for the economic climate? New competition? Demographic changes? Inappropriate performance expectations? Notice that our statement of symptoms — performance below expectations — makes the latter possibility easy to see.

Finally, House adjudicates the fights. He’s willing to make a judgment call but that’s his last resort. He prefers the hypothesis/test method. If the 17 morning espressos are responsible for the headache but not the high blood pressure, then it must be true that the headache will fade as the espressos wear off but the blood pressure will stay high. That approach helps narrow the possibilities. Notice that it doesn’t completely confirm the role of the 17 espressos in the headache. If the patient had sipped the espressos at an outdoor café during rush hour, the headache might be the result of exhaust fumes, not espressos. House would need another test to prove that 17 espressos can cause a headache: administer a new batch of espressos where the air is clean.

Strategy equivalent: think like an experimenter. For example, think of Heckert’s statement as a hypothesis that low prices destroyed Supervalu’s gross margin and didn’t bring in more people. Think of Burd’s hypothesis that they could have done better if they’d cut prices sooner. How can we validate or dismiss those hypotheses? Look at comparable businesses who took those actions. Survey customers. Run simulation-based tests. Be careful how you measure: in a recession, “not bringing in more people” may be success, if other businesses are bringing in fewer people. And watch out for the exhaust-fume problem. What else was going on at the same time that Safeway held its price and Supervalu increased its promotions? The other-things-going-on might have been at Safeway and Supervalu, at competitors, in the economy, and so on, and might have nothing to do with price.

I haven’t conducted any of those experiments; all I’m doing is applying a TV show to a newspaper article so as to make a point and illustrate a technique. Let’s go with my imagination, though, for the sake of that point and technique.

• It wouldn’t be surprising to discover that comparable businesses are disappointed with their results. That would suggest that Burd and Heckert are not doing something uniquely wrong.
• It wouldn’t be surprising to find that customers are cutting back on purchases (buying less and/or buying cheaper) to deal with the economic crisis. That too would suggest nothing unique about Burd and Heckert.
• It wouldn’t be surprising to find that simulations show it’s possible to succeed with steady or lower prices, if the rest of the business adjusts appropriately. (An appropriate adjustment for steady prices might be to reduce capacity to fit a smaller number of customers. For lower prices, an adjustment might be to negotiate lower prices from suppliers or to buy lower-end goods.) We could check to see if Burd and Heckert have made such adjustments.
• We could measure competitors’ results and see if Safeway and Supervalu have gained, held, or lost market share. Their performance might have slid, but if they are outperforming the competition then they might actually be doing well. In other words, the diagnosis might be unrealistically high expectations, and the treatment is to change expectations. They can try to improve performance too, of course, but the role of price cuts is not clear.

There’s much more on this website on the subject of expectations and strategy decision-making, in both the Blog and the Library sections.

Why “again”? Because this isn’t the first time. See also Honey, We Shrunk The Industry, published in Competitive Intelligence magazine, July/August 2009.

A business war game. Five automaker teams: Ford, GM, Hyundai, Toyota, Volkswagen. Three market segments. Two years. One set of consumer judges, one set of investor judges. Fascinating results, again.

The automaker teams were smart and they wanted to win. Yet collectively their decisions subtracted roughly $15 billion of simulated profits from the industry over two years.

It appears that company-centric financial approaches (what are our costs, how much capacity should we mothball) instead of competitive analysis (what will our competitors do, how will consumers respond) led to those problems. In effect, the automaker teams worked by the book, but the book didn’t work.

The good news: Anyone who’d gone through the war game would be less likely to make those mistakes in real life. They would have a competitive advantage.

Background

On September 30, 2009, the front line in the automobile wars could be found in a conference room at Northrop Grumman outside Washington, DC. That’s where a group of 20 strategists converged to war-game the industry. Sponsored by the Greater DC chapter of the Society of Competitive Intelligence Professionals and facilitated by me, the war-game used a simplified version of ACS’ ValueWar™ strategy simulator, customized for the auto industry.

The war-game exercise was designed to demonstrate war-gaming, not to solve the problems of the auto industry. After all, the industry’s problems took decades to build, and it would take our talented strategists more than three hours to fix them all. That said, it was fascinating to see many of the industry’s woes reenacted — and understood — in those three hours. Moreover, the strategists saw for themselves how war-gaming provides a new look at businesses people know well.

War games are not about a consultant or guru dispensing advice to eager supplicants. They are about business strategists living through future scenarios in fast forward and discovering lessons for themselves. They are about strategists making the most of their knowledge, expertise, and creativity. They are about smart people teaching themselves. War games make it possible in a way that conventional strategy development does not.

Why A Business War Game?

Business war games provide a new look at businesses we already know by having us role-play competitors and customers in addition to ourselves, by having us compete as well as compute, by having us encounter action and reaction rather than assume bigger and better. They let us explore and stress-test in a safe environment, where mistakes mean oops instead of ouch.

We can think of business war games as being the highest of three levels of competitive inquiry.

• “What do you think they will do?” This is a basic competitive-analysis question. It explicitly treats competitors as “them.” Strategists, being human, tend to view “them” as less capable than “us.” In practice, answers to this question are often extrapolations of competitors’ past actions.
• “What would you do if you were them?” This question is a major improvement because it encourages the strategist to look through competitors’ eyes. It greatly reduces wishful thinking. Brainstorming or scenario-planning programs may use this question, assuming that they explicitly consider competitors.
• “You are them. You want to win. Go!” This is what happens in a business war game. Strategists don’t only look through competitors’ eyes; they walk in competitors’ shoes. Business war games tap human competitiveness and desires to win. They make tough sparring partners out of genial colleagues. The sparring partners find the flaws in their company’s strategy just as real competitors will. And so strategies get stress tests second only to real life.

The Design Of The Automobile War Game

Teams, segments, judges, and decisions

We divided the war game participants into five automaker teams: Ford, GM, Hyundai, Toyota, and Volkswagen. For the purpose of this war game, these teams/companies competed in three USA market segments:

• Big Tough (roughly SUVs). In the game, this segment was slowly shrinking.
• Slick Style (roughly upscale sedans). This segment was steady.
• Cool Green (roughly eco-friendly vehicles). This segment was slowly growing.

Why five teams? Not to be flip, but three would have been too few for participants to experience the richness of the industry and seven would have been too many to handle in the time we had. Similarly, one segment would have been too few and five too many. Three made for manageable tasks and complexity. Of course, there’s nothing about war gaming that forces those limits. I’ve run war games with eight competitors and with ten segments, and I’ve designed simulation software that can handle dozens of competitors in scores of segments.

We had two judge teams too, representing consumers and investors. The automaker teams would find it difficult to win the war game by creating unidimensional customer-satisfaction or shareholder-wealth strategies. They would have to make real-world tradeoffs.

In addition to excluding some competitors and market segments, we excluded decisions regarding labor, pensions, healthcare, debt service, dealers, government regulations, and suppliers.  We did that for reasons of time and complexity. For the same reasons we limited the game to decisions for pricing, marketing, production, and capacity. That may sound like a short list — real business war games may allow strategists to work with many strategy levers — but it was more than enough to simulate the dilemmas and debacles that real automakers face in real life.

In addition to making pricing and other decisions, the automaker teams presented marketing pitches to the consumer judges and business pitches to the investor judges. The judges’ assessments joined the teams’ decisions in the simulator, which did the arithmetic to estimate demand, sales, profits, and market share.

Calculating outcomes

We used a computer-based strategy simulator, calibrated for the five automakers and the three market segments, to calculate the demand, sales, market shares, and profit consequences of the teams’ strategies. The model took into account common-sense connections like the following. Some of those connections work similarly in other industries, and some don’t.

• Customers have choices. If you invest in marketing or product improvements that customers like, and if you do it better than your competitors, then demand for your autos will rise. If you sit on your laurels, demand is likely to fall.
• Some customers are loyal and will buy again from the same automaker. Other customers are not loyal and will make conscious purchase decisions.
• Price affects both demand and the bottom line. Demand, through customer purchase decisions. The bottom line, by rippling through revenue and costs.
• You decide how much to produce before you find out how much customers want to buy. You cannot sell more than you produce.
• If an automaker cannot satisfy demand, some customers will buy from its competitors (if they have produced enough cars). Other customers will not buy at all if the car they want is unavailable.
• Production capacity costs money. An automaker can save some money by mothballing capacity. However, mothballing will limit what it can produce, and mothballed capacity cannot be brought back on line quickly.
• And more.

There is much more to say about simulation design, customization to industries, and other aspects of business war-gaming, but we won’t say it here. (See other articles and essays on this website.) What’s important is that it is possible to simulate and war-game virtually any industry. My colleagues and I have conducted war games with management in dozens of industries, from airlines to vaccines, on six continents.

About accuracy

All in all, the simulator estimated outcomes pretty well, and definitely better than conventional company-centric analyses. “Estimated” is an important word. We didn’t pretend that this simulator was “accurate” (no analysis of the future is). However, it was definitely realistic and directionally correct, which makes it highly useful for evaluating and stress-testing strategy moves.

Accuracy in future-looking simulations is a fairly complex subject and bigger than I’ll treat here. I’ll just say this. No matter what, strategists will make decisions. The relevant question is not whether a simulation is accurate. The relevant question is whether a strategist can make a better decision with a simulation than without. Oh, and I’ll also say this. There is always a model in strategy decisions. It may be in someone’s head, it may be in a spreadsheet, it may be in a simulator. Choose your model consciously.

Publicly available information and realistic estimates

The war game used publicly available information and realistic estimates as the basis for strategizing and simulating. Nothing proprietary or mysterious, nothing contentious. As in most business war games, the action and the insights come from strategists’ thinking, behavior, assumptions, and decisions, not from decimal points or obscure factoids.

Two rounds (year 1 and year 2) and three hours

The auto teams made decisions about their company’s pricing, marketing, production, and capacity mothballing. They made those decisions for year 1 and we fed their decisions into the simulator. We shared key year-1 results with the teams before they completed their year-2 decisions. They made their decisions for year 2 and we simulated those results too. All, including a debriefing, in three energetic hours (very fast for a business war game).

A level playing field

The five automakers we chose for the war game didn’t start from equal positions and they didn’t have equal resources. We took that into account in the scoring process to ensure that every automaker team had an equal opportunity to win.

It helps for the teams participating in a war game to know there will be a winner because it taps the competitive emotions that affect real-life decision-making. During the business war games we’ve run for real business situations, we’ve seen a company’s own people cheer when they, role-playing the competition, beat their own company! That’s good for the same reason that a boxer wants to practice with a tough sparring partner.

The hard part: Strategic thinking

Competitive strategy is often likened to chess for its complexity and to poker for its competitive interplay. It’s tougher than chess and poker, though, because in competitive strategy the contestants make their moves simultaneously instead of sequentially. You make your strategy decisions before you know your competitors’ strategy decisions.

Take, for instance, this war game.  If you know your competitors will focus on the Slick Style segment and cut production in the Big Tough segment, it’d be reasonable for you to do the reverse. Problem is, simultaneous moves means you don’t and won’t know that. You have to commit to your moves before you know what your competitors will do. (That’s why it’s so helpful to find clues with competitive intelligence and what-if analysis.) It’s a classic and difficult problem, and it affected the automakers in the war game. Here are some of the consequences and the lessons we can draw from them.

Lessons From The War Game

We’ve  run this war game before (June 2009), with SCIP Oregon. We can run it again, too. Contact ACS at info@whatifyourstrategy.com if you’re interested.

Some of the lessons we observed in the previous war game are similar to those from the DC war game, and some are different. The differences are important because they show there’s more than one set of possible strategies and outcomes. The similarities are important because they reveal how strategists commonly think.

These lessons cite various numbers from the simulation. As I said, the numbers are not “accurate.” However, they are sensible and meaningful, and none of the lessons below would materially change if the numbers were off by considerable amounts.

Do not use any information or analysis presented here to make investment or other weighty decisions about the auto industry!

Lesson 1. History isn’t all it used to be

It always happens: strategists tend to think in terms of where they were last year and then adjust up or down. It’s related to a phenomenon that psychologists call “anchoring.” More importantly, it’s due in part to assumptions that there is some kind of static, steady-state, or equilibrium position, assumptions that are reinforced by trend-line and other analyses.

This war game was no exception. Teams went into exquisite calculations with decimal-point precision, figuring out this year in terms of last year plus or minus.

As a result, the Toyota team produced fewer cars than it could have sold. How many? Across the three segments, over 1.2 million in year 1 and 1.3 million in year 2. At prices around $25,000 per car, that means roughly $30 billion in revenue left on the table each year.

The Toyota team wasn’t alone. The GM team was short 581,000 Big Toughs in year 2; there goes $15 billion, more or less. Ford and GM were short 164,000 Slick Style and 260,000 Big Tough cars, respectively, in year 1. Hyundai was low by 138,000 Slick Styles in year 2, which amounts to over 10% of their total sales.

Those shortfalls became unearned bonuses for the automakers who produced more cars than consumers wanted to buy. In other words, automakers who had extra cars to sell were able to sell some to disappointed customers of the out-of-stock automakers. For instance, GM picked up Slick Style sales roughly equal to Hyundai’s shortfall.

Why didn’t the automaker teams know better? Arguably not because they didn’t have enough time, because teams that have hours to develop strategies do the same thing. Definitely not because they were stupid, misinformed, or indifferent, because they were smart, well-informed, and motivated. Why, then? Partly because it’s not the way strategy development usually works. Partly because it’s really hard: it requires analyzing multiple moving parts, including competitors, consumers, production and segment allocations, price, mothballing, marketing spending, and resulting sales. And partly because few people have experience with that kind of systems thinking. All of which are reasons why war games are so surprising and so useful.

Lesson 2. Don’t forget the consumer

In my experience with business war games, teams usually pander to consumers. They promise the earth, they promise the moon, they try to outdo each other by offering shiny baubles and promising elysian delights at bargain prices.

Not this time. Even though I urged the teams several times to produce advertisements to appeal to consumers, and even though I told the teams that reactions from the consumer judges would greatly influence their sales, the teams’ presentations focused almost entirely on the investor judges. There was a minor battle for supremacy of slogans, but that was about it for consumers.

It showed up in the judges’ questions. A consumer judge asked over and over, “Why should I buy a car from you?”

It showed up in the judges’ ratings. On a 0-10 scale, where 0 means not good and 10 means not bad, the average investor ratings were 5.9 in year 1 and 6.9 in year 2. Consumer ratings were 5.1 and 4.8, respectively.

And it showed up in the teams’ results. In year 2, Ford made 1.3 million vehicles it couldn’t sell. They also had low ratings from the consumer judges. The low ratings from the consumers translated into low demand. Low demand in itself doesn’t hurt, but it does when combined with high production and heavy fixed costs for capacity, which is what Ford faced. If Ford had sold those cars, it might have broken even.

GM would have faced a similar fate except it didn’t overproduce quite so much. It only made about 690,000 cars it couldn’t sell, which also represented the difference between a major loss and breakeven.

Volkswagen was hurting too. They had 245,000 leftovers in year 2 (up from 98,000 in year 1). A smaller number, but it was the highest leftover percentage in the game: 51% of the cars made by Volkswagen went unsold.

Meanwhile, Toyota sold out, and Hyundai almost sold out. Toyota was strongly profitable in year 2, and Hyundai was a short drive away from making money. Those were the automakers the consumers liked most and that didn’t over-produce.

Lesson 3. Numbers aren’t only about numbers

Even though we’ve got a bunch of numbers here, the numbers aren’t the point. The point is about the way the teams competed. All of the teams did some things right. All the teams made mistakes, some costing tens of billions of dollars in actual costs or opportunity costs. But the interesting part is where the numbers came from. The numbers — sales, shortfalls, and profits — are, of course, results of doing things right, and of mistakes.

The numbers in the teams’ decisions are expressions of their thinking. The decisions reflected the teams’ predictions and assessments of what moves would work for them. They prioritized their time to focus on investors instead of consumers not because they thought that was a bad idea, but because they thought it was a good idea. They spent $X on marketing because they believed $X was the right amount to spend on marketing, not because they believed $X was the wrong amount. If they thought different decisions would have worked better, they would have made those decisions.

• Speaking about marketing, the teams held their marketing expenses very close to where those expenses began. I don’t know why, other than perhaps taking a budget-oriented perspective: if I overspend I’ll get in trouble, if I underspend I’ll get less next year. (I’ve seen that in other business war games too, where we told participants that they could spend whatever they wanted and all of them kept within a few percentage points of their previous spending.) There were some tweaks, as with prices, but the changes were, in effect, noise.
• The teams focused efforts on the numbers rather than on consumer appeal (i.e., the ads they were encouraged to create), yet the latter could have had tremendous impact. (We could say that the problem wasn’t that some overproduced, it was that they undersold.) Not so dissimilar to what strategists face in real life: huge focus on the numbers and planning, less on blue-sky thinking. It makes incumbents vulnerable to upstarts. Why, after all, is it even possible for upstarts to gain a toehold against incumbents? Incumbents have, or at least can have, all the advantages. The only advantage an upstart can have is thinking differently, and thereby acting differently.
• The calculations some teams recorded for mothballing decisions suggest that the teams were trying to optimize a financial statement rather than to create a competitive strategy. An interesting (at least to me) question is whether it’s better to have a shortfall or to expand capacity to ensure that no demand goes unsatisfied. A strong argument can be made that many woes are due to a desire never to leave money on the table. People can lose a lot of money trying not to leave behind a little bit of money. I’m not saying that a company should go one way or the other. I am saying that there is often a hidden assumption that we should (and can) match supply to demand, and it’s good to notice and challenge hidden assumptions from time to time.

True, the teams didn’t have much time to formulate their strategies. Still, the way they chose to spend their limited time reflects what they believed to be important.

Lesson 4. It’s not enough to be smart

The people participating in our war game were really smart. Those who have participated in other war games were also really smart. The problem is, it’s not enough to be smart. None of us is able to do all the arithmetic in our heads. I’m not saying that any team made mistakes; then again, we all know that no one is always right. Plus, we know that individuals can make rational decisions that, in combination, produce undesired results.

The thing is, when it comes to strategy it’s even hard to tell when an individual or a team is smart. Say you got terrific results. How much of that is because you did something smart and how much is because someone else did something not smart? How much of your decisions were due to luck and how much to thoughtful analysis? How do you know if the same moves will work next year? It can be hard to resist the automatic conclusion that our strategy is working.

We say that people learn from experience. The trick is to get experience where it’s cheap, and to get lots of it. Business war games are cheap, plentiful experience. Cheap, because you’re not playing with real money and real careers. Plentiful, because you can test multi-year strategies in a few hours. With computer simulations, you can test them in a few seconds.

Lesson 5. Define “winning”

The auto team that scored best on profits was Toyota. (That’s a statement about the Toyota war-game team as well as the Toyota brand itself. By no means was Toyota predestined to be big and profitable in the war game.) GM had the biggest overall market share, but Toyota was within rounding error of them. It appears that Toyota had a winning strategy. Appears. However, they were the team that had by far the biggest capacity shortfall in both years. Over two years that team could have sold over 2,500,000 more vehicles. They could have sold roughly $60 billion more.

Other tidbits:

• Toyota gained by far the most share, despite their shortfalls. They wouldn’t have if they had mothballed more capacity.
• Ford took it on the chin. They might not have if they had built less and mothballed more, or if they had impressed the consumer judges more.
• Volkswagen lost a little share. Hyundai gained a bit. GM pretty much held still.

So, did the Toyota team do well (highest profits, biggest share gain) or did they do badly (biggest opportunity lost)? Hard to know. But we do know that it would be hard to see the situation with a conventional spreadsheet, and it was easy to see in a war game with a strategy simulator.

Conversely, the simulation had the Toyota team’s competitors pick up much of what the Toyota team left behind. In effect, the Toyota team’s mistake became a gift that inflated the other teams’ results and that made them look better than their decisions warranted.

Lesson 6. To find better strategies, look again

Of course the teams might have come up with better strategies if they’d had more time. More importantly, the teams would probably have come up with better strategies if we had the time to turn back the clock and try a second round of strategizing.

In the real business war games I’ve conducted, disappointments from the first rounds are essential to getting people’s attention and stimulating people’s creativity. The second set of simulations are where the best strategy ideas come up. Just think: if you’d participated in this business war game, and you knew all these lessons (and more), wouldn’t that help you develop a much better strategy?

Ever see a Fortune 500 company turn on a dime? I have. It’s with the process similar to the one we ran, with those second or third rounds. The key is to get quickly and persuasively to the second or third rounds.

Incidentally, my colleagues and I are strategy agnostic. We don’t come in with a favorite strategy. I for one don’t even care what strategy the company ends up adopting. What I do care about, and I care about this deeply, is the quality of the company’s strategic thinking and decision-making. I don’t care if the right answer for your company is to zig or to zag. I care a lot about helping you find the right answer, whatever it turns out to be.

And In Conclusion

As obvious as those lessons may appear now, they were not obvious before the war game. Corollary: what appears obvious before a war game often turns out to be a really bad idea. That lesson and its corollary are the rule, not the exception, in real business war-gaming.

Business war-gaming and strategy simulations are big subjects and we’ve just scratched the surface. I urge you to learn more about them. Qualitative or quantitative, formal or informal, big or small, facilitated by you or outsiders… business war games help strategists make much better strategy decisions.

Thanks, and congratulations

My thanks to the Greater DC Chapter of SCIP, especially to August Jackson and Jeff Trexel, for inviting me to present the automobile industry business war game.

My thanks to the intrepid strategists who gave their all to the automobile industry for three hours. It was a huge challenge, and you rose to the occasion with intelligence, critical thinking, and humor. You brought great ideas, open minds, and good cheer to the event. Well done.

Congratulations to the Top Strategists on the top-scoring team, Toyota, and to the judges who asked great questions as consumers and investors. Honorable mention to the Hyundai team, which was within a decimal point of Toyota.

This is an announcement for a business war game that ACS will present in Washington, DC, on September 30, 2009. It is in conjunction with the Greater DC chapter of SCIP, the Society of Competitive Intelligence Professionals.

Click here to register for the event. For location, cost, agenda, and contact information, please see the end of this announcement.

We all know we’re in an economic crisis, and we all know that no one knows what we should do. We all know that, difficult as it may be, we still have to make strategy decisions. And we all know that it is a time of opportunity as well as a time of danger.

It’s a perfect time for business war-gaming.

Imagine you are Alan Mulally, the CEO of Ford. Or you are the CEO of GM, Hyundai, Toyota, or Volkswagen. (Of course, business war-gaming is for companies in any competitive industry, not just automobiles.) How do you decide what to do? How do you know what will work? Will you be a hero, or will you fail miserably and publicly? How can your company tap the opportunity and avoid the danger?

Business war gaming helps you plan with greater confidence, generate consensus and action, and improve your chances of success. It helps you generate valuable surprise insights. It helps you think more clearly, anticipate second- and third-order effects, and see the true big picture. Business war games help strategists plan realistically, not optimistically.

Companies such as British Airways, GlaxoSmithKline, Shell, Weyerhauser and many more have worked with Advanced Competitive Strategies to run business war games and make much better strategy decisions.

ACS’ Mark Chussil, veteran of more than 100 business war games, will join us for an interactive, hands-on, half-day session. The highlight of the session will be our own business war game exercise. The automotive industry will be our use case, perhaps the most interesting industry to examine right now because of the perfect storm of long-term trends impacting all at once. Mark led the SCIP Oregon chapter through a similar exercise earlier this year, a program that received very positive reviews. We’re excited to bring Mark and this program to the Greater Washington chapter.

About the speaker

A highly rated, thought-provoking, and entertaining speaker, Mark lectures and consults around the world about strategic thinking, advanced business war games, and computer simulation. Mark is the Founder of ACS. He has received a patent and an industry award for his simulations. He has worked around the world with dozens of Fortune 500 companies on six continents, and he has published extensively about competitive strategy. Mark earned his MBA from Harvard and his BA from Yale.

Location, cost, agenda, and contact information

Northrop Grunman
NGIS Competitive Analysis
7575 Colshire Drive
Mailstop C4S2
McLean, VA  22101
Office # 4232S
(703) 713-4000

SCIP Members – $35.00
Non-Members - $45.00
Students - $25.00 (Non-SCIP Student member, please contact Robyn Reals)

Agenda
8:00AM – 9:00AM: Registration & Networking
9:00AM  11:30AM: Presentation 

Contact Information
Jeff Trexel, Greater Washington DC Chapter Chair, jeff.trexel@infoition.com,
703-556-0027
Robyn Reals, SCIP Education Manager, rreals@scip.org, 703-739-0696 x107

I presume you would like to predict your competitors’ moves better than you do now. Say, for instance, their prices. Let’s work on that, perhaps with a surprise, a jolt, or even a shock as we go along.

We structure today’s harangue around a pricing quiz involving simple math. It’s not a math test, and your enjoyment (or not) of this essay doesn’t depend on whether you enjoy math.

Question 1. Your competitor charges $900 for their product. In month 1, they raise their price by 10%. In month 2, they hold their price steady. In month 3, they cut their price by 10%. In month 4, they raise 10%; month 5, hold; month 6, cut 10%; and so on. What is their price at the end of month 30?

Many people say that the competitor’s price will be $900 at the end of month 30. They figure the 10% increases cancel out the 10% decreases. That’s wrong, as we can see at the end of the first three months. Month 1: $900 + 10% = $990. Month 2: $990 + 0% = $990. Month 3: $990 – 10% = $891. It didn’t cancel out.

Answer to question 1. The competitor’s price at the end of month 30 is $814.

Question 2. What will be the competitor’s price at the end of month 31?

We’ll get to the answer in a moment.

The nice thing about patterns is that we can describe them in numbers. The competitor’s price changes +10%, 0%, -10%, +10%, 0%, -10%. We feel validated because history fits the pattern so well. We feel confident because the statistical “explanatory power” is so high; it’s perfect, in this case. We feel we can predict what happens next, as we do with the great ebb and flow of the tides, the majestic rising and setting of the sun, and the news-cycle persistence of a juicy scandal affecting not me.

We shouldn’t feel validated or confident. At least not yet. And we may not yet be ready to predict the competitor’s price.

Answer to question 2. We can say that the competitor’s price will be 10% higher than $814, or $895. That’s what the pattern says, to n decimal places, where n is a number larger than we require. But what’s much more interesting, if we put down our calculators and put on our thinking caps, is to ask and answer question 3.

Question 3. Why has the competitor’s price been rising, holding, falling?

(Did you ask question 3 before you answered question 2? Give yourself a round of applause if you did. I’ve run this quiz with hundreds of managers in workshops I’ve delivered. Everyone turns right away to math. No one asks question 3, at least not out loud. What good is an unasked question?)

We humans do two things with patterns. First, we see patterns. It’s part of how we figure out how the world works. We test perceived patterns with science, we accept perceived patterns as experience, we even enshrine perceived patterns as superstition.

The second thing we do with patterns is assume they will persist. After all, part of what makes a pattern a pattern is that it (seems to) persist. The tides, the sun, the scandals.

Patterns persist when the underlying forces persist. If the moon were destroyed by a Death Star, Earth’s tides would change. Over time Earth’s rotation (and hence the number and duration of sunrises and sunsets) is changing because… well, it’s complicated, and not germane to your competitor’s price. The length of a scandal is projected to rise and fall with the level of a society’s blaminess. Change the moon, Earth’s rotation, or a society’s blaminess, and the tides, sunrises, and scandals will change.

So what’s controlling your competitor’s price? You wouldn’t ignore the world and set your prices according to +10%, 0%, -10%. Neither would your competitor, unless they have turned over control of their prices to a trend-line equation that will next announce $895 in an eerie mechanical monotone.

Now we’re getting somewhere. What’s controlling your competitor’s price is how they make decisions. A strong pattern in prices is likely to mean that your competitor (or you) have linked prices to something. If their suppliers change prices periodically, then your competitor might change its prices to maintain their margins. If your competitor heavily incents its salespeople to make the numbers each quarter, their salesforce might lobby mighty hard for quarter-end price cuts. Or perhaps that’s how you have priced, and your competitor is predicting and responding to changes they expect you to make!

So which is it? Suppliers? Salesforce compensation? Competitive dynamics? Something else? The point is that people, not calendars, control prices.

Notice that the pattern, whatever’s behind it, makes it hard to know what your competitor will do if you make a change. What would happen if you raise or cut your price when you’re “supposed” to hold? Would they follow? If neither you nor they have deviated from the pattern in a while, you have no data to suggest what they will do. You have to understand your competitor, you have to get inside their heads, you have to model their business, you have to figure out why they price as they price.

Sidebar. I’ve run many, many business war games around the world. They always produce surprises (that’s a good thing) for the management teams that participate. Why? Most of those teams have already analyzed their strengths, weaknesses, opportunities, and threats; what surprises are left? Here’s why I think they get surprised. SWOT analysis mentally places you inside your own company and asks you what you think your competitor will do. A business war game, on the other hand, asks you what you would do if you were your competitor. The latter question explicitly encourages you to look through your competitor’s eyes; the former doesn’t. With the latter you understand them better, you get inside their heads, you model their business, you figure out why they act as they act. End of sidebar.

Intellectually we all know that no one delegates pricing decisions to a calendar. That said, take a look around and see how often people (maybe you and me too) assume patterns will persist. You will see people (maybe you and me too) frame problems in terms of patterns rather than understanding. Listen to pundits, read newspapers, look at analyses.

If you understand your competitors better, you can make better decisions. You can understand better how they will respond if you do this or if you do that. (Do you think they will raise their price to $895 no matter what you do?) Which is, of course, the reason why we try to predict them in the first place. We want to make better decisions for the environment we think we’ll face.

Further reading:
It’s Working!
Precision In, Garbage Out
The Rules
You’re Got the Data. Now What?

ACS announces an upcoming workshop about business war-gaming and a forthcoming article about a business war game for the automobile industry. We also invite strategists to participate in a free and fascinating pricing-strategy tournament.

Workshop
Desperate Competitors and New Rules: Using Business War Games to Compete Effectively During Turbulence. ACS will present this two-day workshop Kuala Lumpur (August 17-18, 2009). The workshop is sponsored by IBN. See below for more about the workshop.

Article
Honey, We Shrunk the Industry: An Automotive War Game. This article will appear in the July/August issue of Competitive Intelligence magazine, published by SCIP (the Society of Competitive Intelligence Professionals). It is based on an ACS blog essay of a similar name.

Pricing-Strategy Tournament
ACS is conducting research on pricing strategy using our Decision Tournament™ technology. We’re using the research in a book about decision tournaments. You can see how your pricing strategies compare to those from hundreds of other strategists! It’s free, fun, serious, fascinating, unique, and confidential. You will develop pricing strategy for businesses in three industries, and you will receive a report detailing the results and implications of your strategy. And if you’re the top pricer, you will be honored on our website! It may not be an Academy Award, but it’s not chopped liver either. Please write to info@whatifyourstrategy.com to enter.

The Desperate Competitors Workshop
How can you compete when everything is changing? How can you know where to commit your resources when the risks are so high? How can you choose new strategies when old strategies have failed and the future is uncertain?

Desperate Competitors and New Rules will challenge the ways you may have thought for a long time. That’s a good thing. What worked in the past may not work when the market is much different, and the companies that succeed in the future will be those who adapt quickly and effectively to that future.

The workshop is not about cheerleading, unsubstantiated magic, miracle strategies, or brow-beating. It is not about uncovering obscure factoids or improving the bottom line by 0.0371 percent. It is not a lecture.

The workshop is not about the past. It is about the future.

Desperate Competitors is about questioning, competing, strategy, and thinking. It is about improving strategists’ skills, applicable every day and starting right away. It is highly interactive, with case studies, lively exercises, simulations, and actual war games. It is designed for people ready to compete effectively by thinking differently.

For more information about the workshop, please write to info@whatifyourstrategy.com.

In this essay we will analyze a difficult problem that’s in the news. It’s about how our beliefs and assumptions guide our analysis of a knotty problem. It’s relevant to anyone who works with data, even if — or perhaps especially if — answers seem obvious.

My home town, New Haven, Connecticut, has been in the news. Not because it’s my home town. Rather, because of a reverse discrimination lawsuit (Ricci v. DeStefano) brought by firefighters against the city. The case is in the news also because Judge Sonia Sotomayor, who’s been nominated for the U.S. Supreme Court, joined her colleagues on the Second Circuit Court of Appeals in a unanimous decision backing the city.

Ricci v. DeStefano is now before the Supreme Court. (Update: for their decision, please see the end of this essay.) I thought it might be nice to bring some cold, rigorous thinking to the hot, emotional case.

The case concerns tests used to promote firefighters to lieutenant or captain. Although the legal arguments focus on which parties have which rights and obligations, the core question (which we might hope is relevant) is whether the tests actually were discriminatory.

Here are the numbers:

• 41 people passed the captain’s exam: 25 white, 8 black, and 8 Hispanic. The city would have to promote the 9 people with the top scores. 7 were white, 2 were Hispanic, none were black.
• 77 people passed the lieutenant’s exam: 43 white, 19 black, and 15 Hispanic. The city would have to promote the 10 people with the top scores. All were white.

The city contends that the apparently too-high percentage of whites being promoted by their test scores is evidence of racial discrimination, and so they threw out the test results. The firefighters say that throwing out the test results reverse-discriminates against those who scored well on a fair test.

I decided to calculate whether the high percentage of white promotions was statistically “too” high. If it is statistically unlikely that whites would do so well and non-whites not so well, we’d have evidence that the tests might have been discriminatory (the city’s position). If the odds are high that it the results could happen by chance or merit, we’d have evidence that the tests were not discriminatory (the firefighters’ position).

Note that analysis can say nothing about the intentions behind the tests. Note also that analysis cannot prove that the tests were discriminatory (or not) in some absolute-truth sense. Still, if the odds strongly favor one side or the other, that should count for something. A reasonable person would and should draw different conclusions about X if the odds of X are 1% and if the odds of X are 99%.
 
I wrote a computer program that looked at every possible way to distribute 41 people in 9 slots (the captain’s exam) and every possible way to distribute 77 people in 10 slots (the lieutenant’s exam). Then, it looked at how many of those possible ways matched the actual racial distribution of the results.
 
There are 350,343,565 possible combinations of 9 winners on the captain’s exam. Of them, 13,459,600, or 3.8%, had 7 whites, 0 blacks, and 2 Hispanics. Another way to look at the results is how many had 7 whites and 2 non-whites. Under that test, 57,684,000 combinations match 7 whites and 2 non-whites, or 16.5%.

In statistical analysis, 5% is a common threshold for “significance;” that is, 1-in-20 odds, a fairly reliable result. (More-stringent analysis uses 1%.) Thus, the 3.8% supports the city’s case, and the 16.5% does not.

Then there’s the lieutenant’s test. There are (this is not an exaggeration or a joke) 1,096,993,404,430 possible combinations of 10 winners out of 77 people. Of that trillion-plus, 1,917,334,783 fit the 10 whites, 0 non-whites outcome. The odds of that are far below 1%; to be exact, the odds are 0.17%. That suggests it was not an accident that 10 whites got the 10 top scores. Why it happened — the test itself, the scoring, self-selection among those who took the test, something else — is a different question, about which neither the analysis nor I make any statement. (I didn’t mention merit as a reason why it happened. We’ll come back to that.) All we can say is that there’s only 1 chance in almost 600 that such an outcome would occur by chance. That’s like guessing a coin toss correctly 9 times in a row: it can happen but you wouldn’t bet on it. Those results support the city’s case strongly.

That said, 1-in-600 odds don’t prove there is discrimination. Those test results could happen by chance, especially if multiple cities use the same test or the same city uses the test multiple times. It’s like winning the lottery if you play enough times or dying on an airplane if you fly enough times. And again, there’s the question of merit.

It gets more complicated. We know that New Haven took pains to create a test that wouldn’t racially discriminate. Assuming that they were sincere and at least partially effective in their efforts, that should raise our confidence that the test results happened by chance (the firefighter’s position), not by discrimination (the city’s). How much should we raise our confidence that the test was not discriminatory? I don’t know. A place to start might be to compare the results of the contested exam with the results of previous tests, or to look at other cities’ test results.

Here’s a different complication: how do we define or discern discrimination? Presumably it would show up as an unfair boost, not unlike steroids, rather than as a blatant gift. Let’s try an experiment. What if, for instance, whites were surreptitiously given slightly higher scores on the tests than blacks or Hispanics? I don’t know how that would be done, but let’s assume that there was a clever way. The average score on the lieutenant’s test for whites was 71.8, for blacks 63.8, for Hispanics 63.6. An unfair boost is one of several possible explanations for that difference. The existence of the difference does not prove the difference was unfair or even statistically reliable, though it begs to be studied more. Regardless, what if we split the difference on the averages and subtract 4 points from every white candidate’s score? How many of the top 10 scorers would be white?

Answer: still 10.

Subtracting 4 points from the whites’ scores is arbitrary, and it could be argued that it introduces clear bias in an attempt to eliminate assumed bias. Even so, 4 points doesn’t change the promotion list. That’s an argument in favor of the firefighters who brought the lawsuit, though it doesn’t prove much.

What if we completely erase the average differences among the groups by subtracting 8 points from the whites’ scores? We’d have 6 white winners and 4 black winners, an argument in favor of the city. And we still haven’t proved much.

Neither the 4-point experiment nor the 8-point experiment proves anything about the presence or absence of discrimination. They merely show the sensitivity of the promotions to presumed systematic bias of a certain number of points. The experiment is about the size of the arbitrary subtraction, not about discrimination. The experiment comes down to whether the experimenter believes that 4 points, or 8 points, or 0 points, or 2.736 points, or 12.345 points, is the right adjustment for the differences in average scores. It’s an analysis based on an assumption. (If there are data that support a real adjustment, that’d be another matter entirely. I don’t know if any such data exist.)

Let’s try another approach. Forty-three whites passed the lieutenant’s exam, along with 19 blacks and 15 Hispanics. Given the much larger number of whites, we’d expect that there would be more variation among them than within either of the other two groups. That happened: there was a wider range of scores among whites. Some of those scores were at the top end. That supports the firefighters’ case.

At last, here’s the merit issue I’ve been promising you. Calculating the 1-in-600 odds started with the implicit and more-or-less invisible assumption that all the people who passed the test got the same score. It’s a direct consequence of treating each of the trillion-plus combinations of winners as equally probable. (Did you spot that assumption? I didn’t until I got pretty deep into my analysis.) In effect, my calculations answered the question “how probable is it that equally qualified people of different races would produce 10 white winners (the lieutenant’s test) or 7 white and 2 Hispanic winners (the captain’s test)?” But how do we know people’s qualifications? That’s what the tests are supposed to reveal. And if the tests reveal merit, then the test results would be right, by definition. But I don’t know if they do (perhaps someone else does); some tests work and some tests don’t. For now, we can only make assumptions.

Presumably the New Haven Fire Department believes the tests measure something of value. On the other hand, presumably no one believes the tests are perfect. So what should we conclude?

Alas, our analysis is not conclusive. If we assume that the candidates were equally qualified, more or less, the 1-in-600 calculation is pretty compelling in favor of the city. Ditto if we assume the tests and scoring were slanted, intentionally or not, toward the white candidates. On the other hand, the wider variation in the larger group, the less-than-overwhelming 1-in-26 odds on the captain’s test, and the city’s previous efforts to ensure fair tests argue in favor of the suing firefighters.

Most important, there’s the question of whether the tests measure merit. If they do, the firefighters’ case is strong. If they don’t, the city’s case is strong.

The bottom line: no definitive answer yet.

Let’s put New Haven aside and up-level the discussion.

When I ran my computer program, I was sure of my conclusion: New Haven is right, the firefighters are wrong. As I wrote this essay, though, I thought about questions my dear readers might ask, and those questions made me think. I questioned my methods, assumptions, and conclusions. I went back and forth as I crunched numbers every way I could imagine short of making this essay my career. So much for cold, rigorous thinking; instead, I got a cold, hard dose of unwanted humility as my conclusion morphed into I don’t know. More data might help, especially information about the validity of the tests. I don’t have those data, and even if I did, there’s only so much time for analysis before we must make decisions. That’s true in government and in business.

That I have not come to a clear conclusion about Ricci v. DeStefano doesn’t mean this exercise (and your faithful reading) has been useless. Quite the contrary; we can come to conclusions about how we come to conclusions. This exercise has:

• Made me less likely to jump to a tempting conclusion based on a short stack of factoids.
• Reminded me that numbers don’t speak for themselves (1-in-600 odds), that analysis reflects basic assumptions (equally probable combinations of winners), and that statistical processes are always at work (wider variations in larger groups).
• Taught me to actively, deliberately look for contrary data and ideas and to keep asking how my analysis could be wrong or incomplete.
• Helped me formulate different ways to solve analytic problems. For example, when there is no clear conclusion, ask a different question. “Which mistake would I rather make” is a good one. “What would a smart reader say” is another.
• Shown me that even though “proof” may be an unattainable standard, we can avoid simplistic answers.
• Proven to me that inconclusive data can lead to more-thorough thinking.

Both sides care about the truth, and we have learned that the data we’ve seen so far are not sufficient to tell us the truth. We have learned that we can debate the promotion patterns and test scores as long as our voices hold out but those data alone do not reveal the truth. We have learned that making a decision relying solely on the data we’ve seen so far will be at least partially the triumph of persuasion or ideology. Finally (and this is both important and exciting), we have learned what else we need — data on the validity of the tests themselves – to make a decision using reason and analysis.

Update. On June 29, 2009, the Supreme Court ruled in favor of the firefighters in a 5-4 decision written by Justice Anthony Kennedy. The Wall Street Journal reported that Justice Kennedy said employers “must show a ’strong basis in evidence’ before ignoring results of employment-related tests.” As we saw in this essay, the evidence was mixed, and whether discrimination had been present or absent, it would have been difficult to prove or disprove.

Further reading
Steven D. Levitt and Stephen J. Dubner, Freakonomics.
Leonard Mlodinow, The Drunkard’s Walk.
John Allen Paulos, Innumeracy.
Jay Russo and Paul Schoemaker, Decision Traps.
Nassim Nicholas Taleb, Fooled by Randomness.
See also Marvelous Techniques.