One-Trial Learning
By Mark Chussil
I love data. I spent 15 years working with the PIMS (Profit Impact of Market Strategy) database alongside academic luminaries. (I was in the room but I was not in their league.) Many years later, I think it’s fun to run and analyze a few billion simulations before breakfast. I love data.
But data isn’t learning, and learning isn’t just about the amount of data.
How much data do we need to learn? One data point gives us an existence proof; that is, proof that something is possible, because it exists. It’s also an anecdote, and the learning that people get from that one data point is properly called a story.
To see why, let’s look at rats.[1]
Imagine a T-shaped maze. A rat enters at the base of the T. It scurries along and soon faces a choice: go one way or go the other.
If the rat goes one way, it gets a piece of delicious cheese. If it goes the other way, it encounters an irrevocably fatal electric shock. (This is a thought experiment. No real or simulated rats were harmed in the writing of this essay.)
That thought experiment parodies a serious idea, one-trial (one data point) learning, proposed by Edwin Ray Guthrie (1886-1959), a psychologist at the University of Washington.[2] The joke, of course, is that a rat that survives hasn’t really learned. It just chose the right path by chance.
That’s all very amusing and/or academic, except not for the rat. As it happens, though, one-trial learning can teach us something about decision-making.
What has a surviving rat learned? The answer is simple: nothing. The rat doesn’t know what would have happened otherwise. (For a different reason, neither does a rat that turned the other way.) The rat has only an anecdote.
If one trial isn’t enough for learning, though, how many does it take for us to know something? Too many. We don’t usually have enough time, let alone money or appetite for risk, to run randomized controlled trials on our competitive strategies. We run test markets on products, we randomize A/B marketing messages, etc., but those experiments don’t go head-to-head with actual competitors.
Fortunately, we can greatly improve the quality of our strategy decisions at costs in time and risk that we can bear. Or, as a data-lover might put it:
Greatly improved decisions ÷ Costs we can bear
= Impressive returns on investment
When the stakes are serious and the possibilities are plentiful, there is simulation, computer-based (e.g., strategy tournaments) and/or role-playing (e.g., business war games). Those techniques can simulate strategies too risky to test in real life, uncover surprises in a safe environment, explore many possible futures, and offer rigor and impartiality. They’re not perfect — nothing is — but they improve the odds of making good strategy decisions. Think of it this way: Odds favor the casino by only a few percentage points when gamblers bet on the roulette wheel (the house edge is between 2.7% and 5.3%), but that’s how casinos make millions.
Traditional alternatives? We accumulate anecdotes. We see which companies succeed and which fail, we make up stories about them, and we believe the stories of their inevitable successes and failures in finding the cheese. It’s better than nothing but it’s not the best we can do. An existence proof doesn’t show something is likely, but it does show something is possible.
Sometimes time is too short for simulation or war-gaming, and the problem is merely whether we’re ready to take advice from anecdotes. For those situations, try a variant on the headline test: Turn the advice around and see if the opposite is possible too. For example, consider “be first to market if you want to win.” Can you win if you’re not first? (In some markets, clearly yes; in others, arguably no.) If you can win without being first, then “be first” is an option, not an obligation. And when you spot new options, you spot new opportunities for competitive advantage.
Unlike the rat, we can tear down the maze walls. It’s a big world out there. Lots of cheese.
Further Reading
Ben Gilad posted a terrific essay on the shortcomings of “data-driven” decision-making. (Ben and I wrote The NEW Employee Manual: A No-Holds-Barred Look at Corporate Life, Entrepreneur Press, 2019.)
Professors Bart De Langhe (ESADE Business & Law School) and Stefano Puntoni (Rotterdam School of Management, Erasmus University) spoke about “Understanding Decision-Driven Analytics” in a 2021 MIT Sloan School webinar.
[1] You think people don’t find rats exciting? Put a rat in a room full of people and see what happens.
[2] For more, see AlleyDog.com, “psychology students’ best friend.”