How the CPI is built and why it never matches your grocery receipt

Your receipt says one thing, the government says another, and you suspect somebody of fibbing. Relax. Odds are, nobody is. The official price number and the bills your own family pays measure different things, and once you know how the official one is built, you can stop shouting at the television and start asking sharper questions of the people who use it.

The number in question is the Consumer Price Index, the CPI, and it quietly runs a great deal more of your life than most people realise. It moves Social Security checks. It has been used to adjust tax brackets. It is written into labor contracts and leases and the rates on certain Treasury securities. A number with that much power over family finances deserves to be understood, not just quoted. So let me take you through the machine.

Who builds it and what it covers

The CPI is produced by the Bureau of Labor Statistics, the same agency behind the jobs report. It is not one index but a family. The headline version, the CPI-U, covers all urban consumers, which is the large majority of the population. A second version, the CPI-W, covers urban wage earners and clerical workers, a narrower group. There are regional versions and an index excluding food and energy, usually called core, which strips out two of the most volatile categories to show the underlying trend. And the BLS publishes an experimental index for households headed by someone aged 62 or older, which we will come back to.

What the CPI measures is simple to state: the change over time in the prices that consumers pay for a fixed collection of goods and services. It includes sales and excise taxes you pay at the register. It does not include income taxes or payroll taxes, and it does not include the price of investments such as stocks. It is a measure of the cost of living day to day, not a measure of the cost of building wealth.

Step one: collect the prices

BLS data collectors gather prices every month from stores, service providers, rental units and doctors' offices across urban areas of the country. They are not asking an average shopper how things feel. They are recording the price of specific items at specific outlets, then going back to the same item next month. The point of tracking the same thing is to separate a price change from a change in what is being sold. If the can of soup is the same can, and the price is up, that counts as inflation. This sounds easy and gets complicated fast, which we will see in a moment.

Step two: decide what the basket weighs

A price index has to answer the question: how much does each price matter? A 10 percent rise in the price of pet rocks should not move the index the way a 10 percent rise in rent does. The answer is the weights, and they come from another survey, the Consumer Expenditure Survey, in which a sample of households report what they actually spend money on. The BLS uses that spending pattern to decide how big each category is, and it refreshes the weights periodically as habits change.

Housing is by far the largest piece, followed by transportation and food, with medical care, recreation, education and communication, apparel and other goods and services making up the rest. That is the average American household's budget, sliced into a pie.

Now watch what happens when you ask a person about that pie.

Nobody lives in the average household

Let me build a hypothetical, labelled as one, with round numbers so you can follow the arithmetic. Suppose the official index has four pieces, and over a year the prices change like this:

The overall rise is each change times its weight, added together: 0.40 times 4 is 1.6, 0.20 times 6 is 1.2, 0.10 times 10 is 1.0, and 0.30 times 2 is 0.6. Total: 1.6 plus 1.2 plus 1.0 plus 0.6, which is 4.4 percent. That is the "official" inflation rate in my made-up world.

Now meet two households. The first lives in a rural county, drives long distances and spends 30 percent on housing, 20 percent on food, 25 percent on gasoline and 25 percent on everything else. Their personal rate is 0.30 times 4, which is 1.2, plus 0.20 times 6, which is 1.2, plus 0.25 times 10, which is 2.5, plus 0.25 times 2, which is 0.5. Total: 5.4 percent. The second is a retired couple whose house is paid off. They spend 25 percent on housing costs such as taxes and upkeep, 25 percent on food, 5 percent on gas and 45 percent on everything else, including medical care. Their rate: 1.0 plus 1.5 plus 0.5 plus 0.9, which is 3.9 percent.

Same economy, same prices, three different inflation rates: 4.4, 5.4 and 3.9. Nobody was lying. The index described a household that does not exist. Real life is a weighted average of everybody, and the average is not you.

The rent that nobody pays

The part of the CPI that confuses people most is housing, and especially owners' equivalent rent. If you own your home, the CPI does not track what you paid for it, and it does not track your mortgage rate. Instead, the BLS surveys homeowners with a question that amounts to this: if someone were renting your home today, unfurnished and without utilities, how much would it rent for? Those answers, alongside actual rents paid by tenants, feed the shelter index.

The reasoning is that a house is both a place to live and an investment. The CPI wants to measure only the first, the service of having a roof over your head, and leave the investment part out. There is logic to that. But it explains a gap many families feel: when home prices and mortgage rates jump, the CPI's shelter component responds slowly, because it is tracking rents, which are sticky, rather than the price you would pay to buy. A family shopping for a house in such a stretch may face a much steeper increase than the index shows. You can see how those payments work in a mortgage worksheet that shows what two points of interest really cost, and why land rules make everything worse in zoning may be the biggest price control you never got to vote on.

Substitution, quality and the problem of the changing basket

A fixed basket has a flaw. If beef doubles in price, people eat more chicken, and an index that insists you still buy the same beef overstates what inflation costs you. Economists call that substitution bias. The BLS has tried several fixes. In the late 1990s it began using a geometric-mean formula for most item categories, which lets consumers shift between items within a category. And in 2002 it introduced a separate index, the chained CPI, which updates the basket to reflect substitution across categories. That chained version grows a little more slowly than the standard one, which turns out to matter a lot in politics. The 2017 tax law switched the inflation adjustment for tax brackets to the chained measure, a change that raises tax bills slightly over time as brackets rise more slowly. If you want the rest of that law, see five changes in the 2017 tax law that reached the kitchen table.

Then there is quality. A television that costs the same as last year's but has a sharper picture is arguably cheaper, not unchanged. The BLS tries to adjust for quality changes, using statistical methods in some categories to separate "this costs more" from "this is better." Critics on the right and the left both complain, with some reason. To a skeptic it looks like the government shaving the number by deciding your new phone is "really" cheaper. To the statistician, ignoring quality would overstate inflation. Both are describing a hard problem honestly, and I would rather you know the problem exists than take either side on faith.

What the Boskin Commission said in 1996

In the mid-1990s, Congress faced a question: a lot of federal spending and taxes were tied to the CPI, so if the CPI was too high, the government was overpaying. The Senate Finance Committee appointed an advisory commission, chaired by the Stanford economist Michael Boskin, to study it. Its report, in December 1996, concluded that the CPI overstated the true rise in the cost of living by about 1.1 percentage points a year. It blamed substitution between items and between stores, new products and quality improvements that the index took too slowly into account.

That finding mattered enormously. Because so many benefits move with the CPI, a lower measured rate saves money for the Treasury, and a commission that tells you the index is too high is, from the point of view of a pensioner, a commission telling you your check should be smaller. The BLS made a number of the technical changes that followed, including the formula change I mentioned. Whether the 1.1 figure was right is still debated by serious economists. The honest conclusion is that the index is a good-faith estimate with known flaws, and that every change to those flaws has winners and losers.

Why Social Security cares

By law, Social Security's annual cost-of-living adjustment is tied to the CPI-W. The Social Security Administration compares the average CPI-W for the third quarter of the year with the same quarter a year earlier, and the percentage increase becomes the raise, announced in the fall and showing up in January checks. If the CPI-W undercounts what seniors actually face, the raise falls short. That is the case advocates make for the experimental index based on households aged 62 and over, which weighs medical care more heavily and has at times run slightly higher than the standard one. The BLS describes it as experimental, and reasonable people disagree about whether Social Security should switch.

For the bigger picture of what the program promises, see How Social Security works and the promise built into its 1935 design, and for what happens if the money runs short, Social Security's repair options laid side by side in one table. The CPI is a quiet lever in both.

Other price gauges you will hear about

The Federal Reserve does not target the CPI. Its stated 2 percent goal is for a different gauge, the price index for personal consumption expenditures, published by the Bureau of Economic Analysis in the Commerce Department. That index uses a different set of weights and a broader scope, and it tends to come in a bit lower than the CPI. When you hear two officials quote two inflation rates in the same week, check which index each used. The story of the Fed's role is in The Federal Reserve in plain English and the levers it really holds.

The shrinking box

Here is one place where the index and your instincts agree. Suppose, as a hypothetical, a box of cereal holds 16 ounces and sells for $4.00, and next month the box holds 14 ounces and still sells for $4.00. The sticker did not move, and a lazy index would record no inflation. The BLS does not do that. It adjusts for the change in quantity, so the price per ounce is what counts: $4.00 divided by 16 is 25 cents an ounce, and $4.00 divided by 14 is about 28.6 cents an ounce, a rise of about 14 percent on that item. A package that quietly shrinks shows up in the index as the price increase it is. Not every quality change is that clean, but this one is, and it is worth knowing that statisticians have an answer for it.

What the CPI cannot tell you

It cannot tell you the price of the specific things your family buys, because it averages across millions of families. It cannot tell you whether your raise kept up, because your income is not in it; for that, see why your raise feels smaller than the number your boss pays out. And it cannot tell you whether the dollar is being debased over decades, which is the larger story in Inflation is the one tax that never needs a vote in Congress. It is a thermometer, not a diagnosis.

Here is the verdict, plain as I can make it. The CPI is neither a fraud nor the gospel. It is a serious, documented, public statistic with real strengths and real weaknesses. The people who tell you it is a hoax cannot show their work, and the people who treat it as the last word on what your family experiences are overselling it. Treat it the way you treat a weather forecast: useful, imperfect, and no substitute for looking out the window.

The bottom line

• The CPI measures the price of a fixed basket for an average urban household, weighted by the Consumer Expenditure Survey. Nobody is that household, so your own rate will differ.
• To find your own rate, list what you spend in each category, weight it by your share of the budget, and multiply by that category's price change, as in the example above.
• Be aware of what the index leaves out: income taxes, investments and the cost of buying a home, which it replaces with an estimate of rent.
• Know which index applies to you. Social Security uses the CPI-W; the Fed watches a different price index; the tax code uses a chained measure.
• When someone quotes an inflation number, ask three things: which index, over what period, and who benefits if it is higher or lower.