Numbers, Circular Reasoning, and Numbers
by Mark Chussil
I didn’t know that Connecticut, my home state, is so big! As reported by CNN.com, the not-yet-decided 2010 gubernatorial race stacks up with, so far, 50% of the vote for Malloy (D), 49% for Foley (R), and 12% for Horner (I). Wow, 111%. Then again, it doesn’t seem quite right that it took Horner 251,494 votes to earn that 12% while Malloy got his 50% with 565,508.
Oddly enough, CNN.com also reports that the undecided senatorial race in Washington has Murray (D) at 51%, Rossi (R) at 49%, and Horner (I) at 12% with 251,494 votes. 112%, even bigger than Connecticut.
Obviously there’s a bug. The details box that pops up retains the last independent candidate even in elections that had only a Democrat and a Republican. It’s not hard to figure out and it’s easy to fix. In fact it was fixed long before I finished writing this essay.
Then there are more-subtle bugs. Here we’ll look at people who switch, 20% liberals, a tremendous success rate, and customer satisfaction. What does that crowd have in common?
People who switch
“People who switched to ______ saved $$$!” Of course they did. They probably wouldn’t have switched if they weren’t going to save money, no matter how articulate the accented lizard or how perky the lady in the whitewashed store. The statement may be true but it’s not helpful. “Honey, it’s going to cost us $1,817 to switch our insurance, but those who switched saved lots so I think we ought to do it anyway.” If you hear that, don’t switch your insurance, switch your honey.
Okay, perhaps the people-who-switched pitch is relatively harmless. You’ll check your insurance premium against what you’re offered elsewhere before you decide, and presumably you’ll ensure that you’re comparing equivalent policies. The ads are designed to wake you out of your lethargy; after all, most of us buy insurance and then sleep on it.
But similar circular reasoning can lead people down questionable paths, especially if they’re depressed or thrilled. Such as after elections.
20% liberals
Pundits raked through vast stacks of numbers immediately after the 2010 midterm elections in the USA. The question was no longer who would occupy the high stations in government where they would soberly, solemnly, wisely, and selflessly help a great nation solve great problems, or where they would prepare for the 2012 elections if that seems more pressing. Once they knew who won and lost the pundits focused on why they won and lost. That answer can guide political leaders as they urgently seek to fulfill their promises to the American people, who had “spoken.”
One of my favorite pundits cited an exit poll in which people could describe themselves as conservative, moderate, or liberal. 20% said liberal. The pundit spoke at length with a guest about the size of that number. They debated whether people are reluctant to say they are “liberal,” since that term has been thoroughly tarred by conservatives, and discussed whether Democrats should more assiduously court moderates.
Not so fast. It was an exit poll. Exit polls are conducted on people who just voted. In 2010 conservatives voted in relatively great numbers and liberals in relatively small numbers; that, by definition, is why the Republicans did well. We may know that 20% of those who did vote consider themselves liberal, but we don’t know the percentage of liberals among people who could vote.
A more-useful question would be to look at the percentage of people in the general registered-voter public who describe themselves as liberal and contrast that to 20%. (Might also be useful to test the word “progressive” instead of “liberal.”) It makes a difference if Democrats lost because there are too few liberals or if they lost because there are plenty of liberals who didn’t go out to vote.
Tremendous success rate
There’s an organization that did better than the Republicans in 2010. I don’t know its name, and, to be honest, I’m not even certain it exists. I heard about it when I attended a seminar. But the story is good, and at least in theory such an organization ought to work.
The organization runs programs to help people make changes in their lives. On the first day of the program the facilitator tells participants to make a simple change in their homes that evening, such as reversing a roll of paper towels so that it feeds from the top (or bottom) instead of from the bottom (or top).
The next day, the facilitator asks who reversed their roll of paper towels. Those who left their paper towels unchanged are thanked, given a full refund, and politely asked to leave. The reason: if they’re unwilling to make a trivially simple change, they are not prepared to make bigger changes. They’re encouraged to return when they’re ready.
So the organization gets a tremendous success rate because they filter out those who do the equivalent of buying health-club memberships and never going to the gym. This clever approach may actually be a good thing because it quickly reinforces the behavior it seeks to instill, and it eliminates people whose attitudes might pull down the group. Still, it does not tell the potential customer what he or she really wants to know: what results will I get if I faithfully complete that program versus some other program.
Customer-satisfaction surveys
Having gotten this far, you may be able to predict the issue we’ll discuss regarding customer-satisfaction surveys. Take a moment to think about it.
Remember the exit polls as you think about it.
You do customer-satisfaction surveys with your customers.
Customer-satisfaction surveys can provide tremendously valuable insight into people’s experiences with a product or service. They don’t say much, though, about what made a person become a customer in the first place. To learn that, you need to contrast what was different about people who became customers (what did they want, what did they believe, etc.) from people who decided not to become customers. Ideally you’d split the latter group into those who decided to buy from your competitors and those who decided not to buy at all.
Here’s another way to think about it: customer satisfaction concerns the previous purchase, and for strategy and forecasting you need to worry about the next purchase. I was very happy with my trusty, powerful H-P calculator (I am dating myself, and all too accurately), and I’d give H-P fabulous customer-satisfaction scores. That said, I will not buy another calculator from them, or from anyone. I’ve got calculators in my computers, iPad, cell phone, and maybe in the smoothie maker I made famous in Motor Swilling Forbidden.
What they have in common
Switches, liberals, change, satisfaction. What do they have in common?
All concern people trying to use information to make better decisions. All are susceptible to circular reasoning (our customers love us, that’s why they’re our customers). All show how critical thinking can prevent mistakes.
- It doesn’t matter if those who switched insurance saved $450.22 or $453.87. It doesn’t matter if 20% of those who went to the polls were liberals, or 20.3%, or 19.735%. And so on. Whether the numbers we have are right is usually the smaller issue. The larger issue is whether we have the right numbers.
- It doesn’t take rocket science to avoid getting misled. Suggestion: state out loud exactly what a number represents and what’s missing, then see if it’s useful for you. When you hear “people who graduated from this program had tremendous success,” ask not only about the tremendous success but also about the people who. Who were they? How might they differ from you? How might they differ from those who did not graduate? How do they differ from those who did not enter the program?
- If you don’t have the right numbers but you’ve got something close, try to make an estimate. Getting close fast can let you make a reasonable decision fast. But beware: using shortcuts to get a reasonable number quickly is one thing, and using shortcuts to eliminate scenarios is quite another. See The How-Likely Case.
“USA Today has come out with a new survey — apparently three out of every four people make up 75% of the population.” — David Letterman
In related news
Kudos to Newsweek’s Sharon Begley, for “Wanted: BS Detectors” (“bs” = “bad science”), in the October 28, 2010, issue of the magazine. “Understanding what counts as evidence should … trump memorizing the structural formulas for alkanes.”
Update, April 14, 2011. “Tasting Wine: A Coin Toss?“, from the Republic of Math. A random person might correctly judge whether a wine is swell or swill with coin-toss odds. That’s not the same thing as saying that a given person has only coin-toss odds of judging multiple wines correctly.
Update, January 5, 2012. Kudos to The Wall Street Journal’s Carl Bialik, “The Numbers Guy,” for “A World Full of Alarming Traces.” He discusses why the finding that 9 out of 10 baby-changing (odd term, “changing” your baby) stations test positive for cocaine powder, while not a good thing, may not be as alarming as it sounds.