The Track-Record Fallacy

The Track-Record Fallacy

By Mark Chussil

Advantage in the skill of competing is the ultimate com­petitive advantage.

How do we know when someone is skillful at competing? Reflexively, unconsciously, and seem­ingly inevitably we turn to the person’s track record. A person with a string of competitive suc­cesses is more likely to be competitively skillful, we figure, than someone with a string of com­petitive failures.

The thing is, it’s hard to tell the difference between luck and skill. We all know luck plays a part. (Bad luck, anyway: we fail when we’re unlucky. Success always comes from skill, we assure our­selves.) That’s why we hesitate to call a person a genius after one good move or a dud after one bad move. But with a reasonably long track record it’s easy to fall into the track-record fallacy, the business equivalent of the prosecutor’s fallacy.

Stop! Do not let your brain recoil in fear of probabilities and actual logic! This is your chance to improve your skill at competing relative to the weaker souls who rely on lists of five magic se­crets of success.

The prosecutor’s fallacy

Imagine that a person has been arrested for murder. (Such a cheerful essay. First I threaten you with probabilities and logic, now I present a major felony.) Police searching the murder scene recover a bit of DNA found near the victim, match it to a person in their database, and make an arrest. At trial, the prosecutor reports that the odds are one in a million that the accused would have that DNA, so the accused is extremely likely — 999,999 to one — to be guilty.

But wait, cries a citizen wise to the prosecutor’s fallacy, that’s not enough. There are ten million people in this city. With one in a million odds, that means about ten people in the city would match that DNA just by chance. If the DNA evidence is all you’ve got, then all we know is that the accused has a one-in-ten chance of being the murderer. If you have more evidence then your analysis would be correct: you’d show the odds that a per­son who fits the known facts of the case also has a DNA match. But without anything else, all we know is that you have picked up one of the ten DNA matches, and by definition almost all of those people are innocent.

Notice that the danger of the fallacy goes down if the city is smaller or the odds of the DNA match are longer. If you’re living in an Agatha Christie isolated-island mystery with only ten people, a murder weapon, and a DNA lab, and one person matches the DNA with one in ten mil­lion odds, then you’ve probably found the perp. Don’t turn your back.

The prosecutor’s fallacy and track records

Say you’re looking for an expert to help you pick stocks. (We’ve arguably put felonies behind us.) There are many thousands of people legally engaged in that activity. One proudly shows you that she has made ten good stock picks in a row. Assuming that the odds of a good stock pick are roughly 50/50[1], that means she’s compiled a one-in-1,024 record.[2]

If there are 10,000 stock-picking professionals, then we’d expect 10,000 ÷ 1,024, or just about ten, to compile a ten-hit record by chance alone. All her track record tells you, by itself, is that she might be one of those ten.

The way you gain confidence (but never get proof) that’s she’s got talent rather than luck is by finding out how she makes her stock picks. That’s the equivalent of the additional evidence from the prosecutor. For example:

  • If she picks yes or no depending on that day’s random pick of a letter from the Code of Hammurabi, figure luck. If she picks based on cogent competitive analysis of the com­pany’s prospects, assume at least some talent.
  • If she talks about certainties, streaks, or something “due” for a change, steer clear.[3] If she talks objectively about multiple data sources and probabilities, steer near.

Best of all would be if she shares stock picks in advance and lets you see how they do. That side­steps the track-record fallacy because it lets you see whether this person can make good picks rather than have you found a person who made good picks. Not perfect, but about as good as it gets.

Avoiding the track-record fallacy

Now, substitute competitive-strategy decisions for stock-picking decisions. Keep the idea of track records.

There are two places where you want to avoid the track-record fallacy. One is track records you use to judge others’ competence. The other is how you think about your own track record.

There are millions of businesses and business units out there, and tens of millions of people af­filiated with the track records of their performance. People make relatively few highly conse­quential business decisions during a career; it’s not like honey, what do you want for dinner. So if we have ten business decisions, one in a thousand odds, and millions of people making those decisions, we’d expect thousands of people to get a fine track record just by chance. We’d ex­pect multiple people to get twenty right decisions, no errors, one in a million odds.

We can never be sure that luck played no part in success. A skillful competitor, though, will take steps not to be blinded by luck. For example:

  • Learn about cognitive biases to think more rigorously and objectively.
  • Become numerate to separate cause and effect from correlation.
  • Use tools (e.g., war games) to put yourself in competitors’ and customers’ shoes.
  • Like the prosecutor, look for more evidence.
  • Like the stock-picker, look for sensible thinking.
  • Beware: people share only positive track records.
  • Gauge wisely: raising sales 25% sounds good, but not if the market grew 30%.
  • Be careful with judging results: was performance too low or were goals too high?

True skill also requires the wisdom to accept that odds are not outcomes. No one succeeds all the time, if by “all the time” we mean a non-trivial number of events and if by “succeed” we mean while playing fairly. Avoid the flip side of the fallacy that says an undesired result neces­sarily implies a bad move.

Just remember: there’s a reason why financial-disclosure statements say past performance does not guarantee future results.

If you want a guarantee, buy a toaster.
Clint Eastwood (BrainyQuote.com)

[1] That’d be true under certain conditions. Those are the conditions I am assuming for this illustration. We could adjust the numbers for different odds; the points would be the same.

[2] There are 210, or 1,024, possible combinations of hits and misses in ten picks. With 50/50 odds on each pick, the 1,024 scenarios are equally probable. Only one combination hits ten out of ten.

[3] “Due” is a probability fallacy when we’re dealing with independent events. When you flip a coin it doesn’t remember how it landed before and try to balance the heads and tails. And if we’re not dealing with independent events, then it’s unfair to count them as separate wins in the track record.

Share This Comment

Comments
No Comments