The Track-Record Fallacy
By Mark Chussil
Prowess at the skill of competing is the ultimate competitive advantage.
How do we know when someone is skillful at competing? Reflexively, unconsciously, and seemingly inevitably we turn to the person’s track record. A person with a string of competitive successes is more likely to be competitively skillful, we figure, than someone with a string of competitive failures.
The thing is, it’s hard to tell the difference between luck and skill. We all know luck plays a part. (Bad luck, anyway; we fail when we’re unlucky. Success always comes from skill.) That’s why we hesitate to call a person a genius after one good move or a dud after one bad move. But with a reasonably long track record it’s easy to fall into the track-record fallacy, the business equivalent of the prosecutor’s fallacy.
The prosecutor’s fallacy
Imagine that a person has been arrested for murder. Police searching the murder scene recover a bit of DNA found near the victim, match it to a person in their database, and make an arrest. At trial, the prosecutor reports that the odds are one in a million that the accused would have that DNA, so the accused is extremely likely — 999,999 to one — to be guilty.
But wait, cries a citizen wise to the prosecutor’s fallacy, that’s not enough. There are ten million people in this city. With one in a million odds, that means about ten people in the city would match that DNA just by chance. If the DNA evidence is all you’ve got, then all we know is that the accused has a one-in-ten chance of being the murderer. If you have more evidence, then your analysis could be correct: You found a person who fits the facts of the case and who has a DNA match. But without anything else, all we know is that you have picked up one of the ten DNA matches, and by definition at least nine of those people are innocent. Perhaps all ten. It’s possible the murderer came from out of town.
Notice that the danger of the fallacy goes down if the city is smaller or the odds of the DNA match are longer. If you’re living in an Agatha Christie isolated-island mystery with only ten people, a murder weapon, and a DNA lab, and one person matches the DNA with one in ten million odds, then you’ve probably found the perp. Don’t turn your back.
The prosecutor’s fallacy and track records
Say you’re looking for an expert to help you pick stocks. (We’ve arguably put felonies behind us.) There are many thousands of people legally engaged in that activity. One proudly shows you a list of ten good stock picks in a row. Assuming that the odds of a good stock pick are roughly 50/50[1], that means a one-in-1,024 record.[2]
If there are 10,000 stock-picking professionals, then we’d expect 10,000 x (1 ÷ 1,024), or just about ten, to compile a ten-hit record by chance alone. The expert might be genuinely skillful or might be one of the lucky ten. Which is it?
The way you gain confidence (but not quite proof) that the expert’s got skill rather than luck is by finding out how the expert picks stocks. That’s the equivalent of the additional evidence from the prosecutor. For example:
- If the expert picks yes or no depending on that day’s random draw of a letter from the Code of Hammurabi, figure luck. If the expert relies on cogent competitive analysis of the company’s prospects, assume at least some skill.
- If the expert talks about certainties, streaks, or something “due” for a change, steer clear.[3] If the expert talks objectively about multiple data sources and probabilities, steer near.
Best of all would be if the expert shares stock picks in advance and lets you see how they do. That sidesteps the track-record fallacy because it lets you see whether this person can make good picks rather than have you found a person who made good picks. Not perfect, but about as good as it gets.
Avoiding the track-record fallacy
Now, substitute competitive-strategy decisions for stock-picking decisions. Keep the idea of track records.
There are two places where you want to avoid the track-record fallacy. One is track records you use to judge others’ competence. The other is how you think about your own track record.
There are millions of businesses and business units out there, and tens of millions of people affected by the track records of their performance. People make relatively few highly consequential business decisions during a career. So if we have ten business decisions, one in a thousand odds, and millions of people making those decisions, we’d expect thousands of people to get a fine track record just by chance. We’d expect multiple people to get twenty right decisions, no errors, one in a million odds.
We can never be sure that luck played no part in success. A skillful strategist, though, will take steps not to be blinded by luck. For example:
- Learn about cognitive biases to think more rigorously and objectively.
- Become numerate to separate cause and effect from correlation.
- Use tools (e.g., business war games) to put yourself in competitors’ and customers’ shoes.
- Like the prosecutor, look for more evidence.
- Like the stock-picker, look for sensible thinking.
- Remember that people voluntarily share only positive track records.
- Gauge wisely: raising sales 25% sounds good, but not if the market grew 35%.
- Be careful with judging results: was performance too low or were goals too high?
True skill also requires the wisdom to accept that odds are not outcomes. No one succeeds all the time. Avoid the fallacy that says an undesired result necessarily implies a bad decision.
Just remember: there’s a reason why financial-disclosure statements say past performance does not guarantee future results.
If you want a guarantee, buy a toaster.
Clint Eastwood (BrainyQuote.com)
[1] That’d be true under certain conditions. Those are the conditions I am assuming for this illustration. We could adjust the numbers for different odds; the points would be the same.
[2] There are 210, or 1,024, possible combinations of hits and misses in ten picks. With 50/50 odds on each pick, the 1,024 scenarios are equally probable. Only one combination hits ten out of ten. Also, only one combination misses ten out of ten, but a person might not survive in the job that long.
[3] “Due” is a probability fallacy when we’re dealing with independent events. When you flip a coin it doesn’t remember how it landed before; it doesn’t try to balance the heads and tails. And if we’re not dealing with independent events, then it’s unfair to count them as separate wins in the track record.