How to Spot a Misleading Statistic Before It Spots You, With Worked Examples

Lies, damned lies and statistics, runs the old line, usually blamed on a dead Victorian. Statistics got a bad rap. Most bad statistics are not lies at all. They are true numbers used badly, which is much harder to catch, because when you say "that figure is wrong," the person answers "no, it's accurate," and he is right, and you look foolish.

The cure is not a degree in mathematics. It is a short list of questions, asked in the same order each time, that catches nearly every trick. Below is a worksheet you can reuse. Each check has a rule, an example with invented figures (every number in this post is hypothetical, built to show the method, and none describes any real town, state or poll), and the arithmetic written out so you can follow it with a pencil.

Both parties use these tricks. Your own side will hand you a misleading number at some point, and the point of a worksheet is that it does not care which side that is. If you want the companion piece on judging the outlet that carries the number, it is How to Evaluate Any News Source With a Twelve-Point Checklist.

Check 1: percent change or percentage points?

The rule. When a number that is already a percentage changes, there are two ways to describe the move, and they sound wildly different. A percentage point is the plain subtraction of one percentage from another. A percent change is that difference divided by where you started.

Worked example. A hypothetical state raises its sales tax rate from 5 percent to 6 percent.

• Percentage points: 6 minus 5 equals 1 point.
• Percent change: (6 minus 5) divided by 5 equals 1 divided by 5, which is 0.20, or 20 percent.

Both are true. Now see what each headline does. "Sales tax rises by just one point" sounds trivial. "State hikes sales tax 20 percent" sounds like a revolt. The thing the shopper cares about is in the middle: on a $200 purchase, 5 percent tax is $10 (200 times 0.05) and 6 percent tax is $12 (200 times 0.06). The tax bill rose by $2, which is exactly 20 percent of $10.

A second case to practice on, with a hypothetical unemployment rate that goes from 4 percent to 5 percent: that is 1 point, and (5 minus 4) divided by 4 equals 0.25, a 25 percent rise in the rate.

Ask: Is this a change in points or a change in percent? Which one is the writer using, and is it the one that makes the story louder?

Check 2: mean or median?

The rule. The mean (the average people usually mean) adds everything up and divides by the count. The median is the middle value when you line everything up in order. When a few huge values sit at one end, the mean gets dragged toward them and stops describing a typical case.

Worked example. A hypothetical town has ten households. Their annual incomes, in thousands of dollars, are 40, 45, 50, 50, 55, 60, 60, 65, 70 and 405.

• Add them: 40 + 45 = 85; plus 50 = 135; plus 50 = 185; plus 55 = 240; plus 60 = 300; plus 60 = 360; plus 65 = 425; plus 70 = 495; plus 405 = 900.
• Mean: 900 divided by 10 equals 90, so $90,000.
• Median: with ten values, the middle falls between the fifth and sixth. The fifth is 55 and the sixth is 60. The median is (55 + 60) divided by 2, which is 57.5, so $57,500.

A reporter can write "The average household in town earns $90,000" and be telling the truth. But nine of the ten households earn less than $90,000, and the typical one is closer to $57,500. The one household at $405,000 pulls the mean up by $32,500 (that is 90 minus 57.5).

This cuts both ways. A politician arguing against a tax can quote the mean and a politician for it can quote the median, and both can be honest. A fair report gives you both.

Ask: Is it the mean or the median? If the two differ much, why?

Check 3: total or per person?

The rule. A total can grow while the amount per person shrinks, and the reverse. Whenever a population, a student body or a customer count changes, a total number tells you only half.

Worked example. A hypothetical town's school budget goes from $10 million to $12 million, while enrollment goes from 2,000 students to 3,000.

• Total change: (12 minus 10) divided by 10 equals 0.20, so up 20 percent.
• Per student before: $10,000,000 divided by 2,000 equals $5,000.
• Per student after: $12,000,000 divided by 3,000 equals $4,000.
• Per student change: (4,000 minus 5,000) divided by 5,000 equals minus 0.20, so down 20 percent.

The superintendent who says "we increased the budget by 20 percent" and the critic who says "we cut spending per student by 20 percent" are both stating accurate numbers. Neither one is lying, and neither one is giving you the whole picture. A voter deciding on a levy needs to know which figure answers the question he is asking: is the town paying more, or is each child getting more?

The same applies to the federal budget, to the national debt and to anything else that grows with the country. See twelve questions about the national debt, answered straight for what happens when you compare debt to the size of the economy and not just to last year's number.

Ask: Per what? If the group being counted changed in size, is the total the right number?

Check 4: where does the chart start?

The rule. The starting point of a comparison can create a trend or erase one. Whoever picks the start year picks the story.

Worked example. A hypothetical price index for a basket of everyday goods reads 100 in Year 1, 80 in Year 3 and 95 in Year 9.

• Start at Year 3: (95 minus 80) divided by 80 equals 15 divided by 80, which is 0.1875, or up 18.75 percent.
• Start at Year 1: (95 minus 100) divided by 100 equals minus 0.05, or down 5 percent.

Same data. Same end point. One speaker says prices "soared nearly 19 percent," the other says prices "are lower than they were in Year 1." If one of them chooses Year 3 because it was the low point, and the other chooses Year 1 because it was the high, each is being truthful and each is steering you.

Fair practice is to show the whole run of years and to say why the starting point was picked. If a chart begins at the single best or worst year for the speaker's argument, ask what the line looked like before it.

A note on charts: also check whether the vertical axis starts at zero. A line that runs from 95 to 100 on an axis that begins at 94 looks like a rocket launch. On an axis that begins at zero it looks like a flat table.

Ask: Why that start year? What does the picture look like if I move the starting point?

Check 5: how many people were asked?

The rule. The smaller the sample, the wider the margin of error. A rough rule for a simple random sample: at the usual 95 percent confidence level, the margin of error in a yes-or-no question is about 1.96 times the square root of (p times (1 minus p)) divided by n, where p is the share who said yes and n is the number surveyed.

Worked example. A hypothetical survey of 20 people finds 12 say yes.

• Share saying yes: 12 divided by 20 equals 0.60.
• p times (1 minus p): 0.60 times 0.40 equals 0.24.
• Divide by n: 0.24 divided by 20 equals 0.012.
• Square root of 0.012 is about 0.1095.
• Multiply by 1.96: about 0.215.

So the margin of error is about 21 or 22 percentage points either way. The headline says "60 percent," and the truth lies somewhere between roughly 38 and 82 percent. That is not a finding. It is a shrug with a percentage sign on it.

Now take a hypothetical poll of 1,000 people with the same 60 percent share.

• 0.24 divided by 1,000 equals 0.00024.
• The square root of 0.00024 is about 0.0155.
• Multiply by 1.96: about 0.030.

The margin of error is about 3 points. Fifty times as many respondents (1,000 divided by 20 is 50) narrowed the range from over 40 points wide to around 6 points wide. The reason: the margin shrinks with the square root of the sample, not with the sample itself.

Two warnings. The formula assumes a truly random sample. A web poll that anyone can click, or a survey of people who showed up to a rally, has no margin of error worth quoting, because the sample picked itself. And ask how the question was worded, because "Do you support cutting waste?" and "Do you support cutting this program?" are different questions that produce different answers.

Ask: How many were asked? How were they chosen? What was the exact question?

Check 6: does one thing cause the other, or do both follow something else?

The rule. Two numbers that move together are correlated. That does not mean one causes the other. Often both are driven by a third factor, most commonly size.

Worked example. Two hypothetical towns. Town A has 50,000 people, 25 churches and 40 bars. Town B has 10,000 people, 5 churches and 8 bars. A lazy analyst notes that the town with more churches also has more bars and publishes "More churches, more bars!" Check the cause with a rate:

• Town A churches per 10,000 people: 25 divided by 5 (since 50,000 is 5 groups of 10,000) equals 5.
• Town A bars per 10,000: 40 divided by 5 equals 8.
• Town B churches per 10,000: 5 divided by 1 equals 5.
• Town B bars per 10,000: 8 divided by 1 equals 8.

The rates are identical. The two counts rose together only because Town A is five times larger (50,000 divided by 10,000 is 5). Size was the hidden factor.

A second sort of error flips the arrow. Suppose a hypothetical report finds that the bigger the fire, the more fire trucks at the scene, and concludes that fire trucks cause damage. The trucks came because the fire was big. Whenever you see "A goes with B," ask three questions: could B be causing A? Could something else cause both? Could it be chance in a small group?

Ask: Is this a rate or a count? What third factor could drive both? Which way does the arrow point?

A bonus trap: the tiny base

One more trick deserves its own warning, because it can be combined with all the others. A big percentage change on a tiny starting number is not a big change.

Worked example. A hypothetical hardware store reports 1 shoplifting incident in one year and 3 in the next.

• Percent change: (3 minus 1) divided by 1 equals 2.0, or up 200 percent.
• Absolute change: 3 minus 1 equals 2 incidents.

"Shoplifting triples at local store" is accurate. It is also two incidents. If the same store sells 50,000 items a year, three incidents is 3 divided by 50,000, or 0.006 percent of items, and one incident is 0.002 percent. Always ask for the raw count next to the percentage. A percentage with no count behind it is a magician's hand: it is moving so you will not look at the other one.

A practice problem, with the answer

Try this one before reading the answer. A hypothetical candidate says: "Under my opponent, the county's road budget rose 30 percent, from $10 million to $13 million, while the number of miles of county road stayed the same. Meanwhile potholes reported rose from 200 to 260."

1. Check the percent: (13 minus 10) divided by 10 equals 0.30. The 30 percent is right.
2. Per mile: the miles did not change, so per-mile spending also rose 30 percent. Per person is unknown, so ask for the county's population change.
3. Potholes: (260 minus 200) divided by 200 equals 0.30, also 30 percent. So reported potholes rose by the same percentage as the budget.
4. Cause: reports are not potholes. More reports could mean a better reporting app, a wet winter or more cars, and a budget that rises 30 percent while reports rise 30 percent does not show that money causes potholes. Ask what the count of reports means before you build an argument on it.

The honest conclusion is "I don't know yet, and here are the three numbers I would need." That sentence is not a weakness in an argument. It is how you get the other person to take you seriously.

The worksheet, to copy

Next time a number lands in your inbox, copy this and fill it in. It takes about five minutes.

If a claim passes all six, it is probably sound. If it fails two, hold it at arm's length. If you cannot find the information to answer a question, that is an answer too.

Putting it to use at home

These checks matter most when a number touches your wallet. The sort of figures that cross a kitchen table are about taxes, prices and budgets. If you want practice on real shapes of problems, seven tax myths that scare working families and cost them money and three households and one grocery bill, budget scenarios when prices jump are good places to apply Checks 1 and 3. Inflation is the one tax that never needs a vote in Congress is a good place to see why the start year in Check 4 matters so much when prices are the subject. For arguments with relatives who have a chart of their own, How to Argue Politics at the Holiday Table and Still Be Invited Back has scripts that go well with this worksheet.

One last thing. You do not have to win the number war. You just have to be the person at the table who asks, politely, "Compared to what?" That question has killed more bad arguments than any fact in history.

The bottom line

• Most bad numbers are true numbers used badly. The fix is a repeatable check, not a distrust of all numbers.
• Ask six things every time. Points or percent, mean or median, total or per person, which start year, how big the sample, and cause or coincidence.
• Demand the denominator. A figure with no "per what" and no "compared to when" is a slogan.
• Run it on your own side first. The worksheet is only worth using if it also catches the numbers that flatter your argument.
• Say "compared to what?" At the table, in the comments and at the town hall, it is the cheapest and most effective question you can ask.