Every Metric Is a Proxy: Goodhart's Law for AI Systems

Mostafa Dhouib·September 4, 2026·7 min read

Every Metric Is a Proxy: Goodhart's Law for AI Systems

The short answer. You cannot personally read forty thousand answers, so you appoint one number to go check the truth and report back. That number is a messenger, and every dashboard figure is one. The failure is always the same shape: the messenger returns and says fine, everyone stops looking, and nobody asks whether it actually checked the thing it was sent to check. Then, once the number becomes the goal, it starts moving for reasons that have nothing to do with what you wanted.

A team watches a quality score. It reads 94, green, and it has read 94, green, for months. Meanwhile the system is handing people confident, cited, completely wrong answers.

Both of those things are true at the same instant, and there is no contradiction between them. The number was never measuring what everyone assumed it was measuring.

This is not carelessness. Watching a clean quality number is exactly what a competent team is supposed to do. That is what makes it worth understanding structurally rather than filing under somebody else's sloppiness.

The messenger

Start from why the number exists at all.

You care whether the answers are true. You cannot read forty thousand of them. So you appoint a messenger: one number, sent out to check the truth on your behalf and report back.

You do not care about the number. You care about the thing it went to check. And the entire failure mode of dashboards is that the messenger comes back, says fine, everyone stops looking, and nobody ever asks whether it actually checked the thing it was sent to check.

Every figure on a wall is this. Accuracy is a messenger for "people get correct answers at work." Uptime is a messenger for "the service is useful." A quality score is a messenger for "we can trust the output." Each is connected to the real thing by a chain of assumptions, and the chain is where the failures live.

The habit worth building is small and cheap: for each number you rely on, write down in one plain sentence what it is standing in for. Half the time the problem is obvious the second it is in plain words, because nobody had ever bothered to write it down.

Four ways a messenger lies

The messenger does not lie by being wrong. It lies by being right about the wrong thing.

It measures a different room. A 95 percent accuracy figure computed on a test set drawn from the same source as the training data is not measuring whether the system works. It is measuring how much the test set resembles the training set. Both are real measurements. Only one was the question.

It averages independent things. A quality score that blends retrieval quality and generation faithfulness into one figure describes neither. A thermostat wired to a freezing room and a burning room reports a pleasant average that is technically correct and completely useless, and it hides the room that is on fire. An average always throws information away; the only question is which information, and here it throws away exactly the piece you needed.

It hides a small slice. A number of 88 percent across document types can conceal a category at 41 percent, because that category is a small share of volume. The average barely moves. For the people in that slice, the system is wrong more than half the time.

It measures the effort, not the work. An iteration counter tells you how much you have spent, not how much remains. A fuel gauge is accurate, honest, and cannot distinguish arriving from running out of gas.

Then it stops being a measurement and becomes a target

The second half of the problem is what happens once people are held to the number.

Goodhart's law is usually stated abstractly and it is very concrete in practice: the moment a number becomes the goal, people find a way to make it go up, and that way frequently has nothing to do with what you wanted.

The clean version is a student studying by memorising last year's answer key. The score climbs sharply. Nothing was learned. The measurement was valid right up until it became the objective, at which point it started measuring something else.

In an AI system it looks like this. The metric line keeps climbing, quarter after quarter, and the actual quality of the thing rises for a while and then quietly turns and heads down. The gap between the two opens slowly. Nobody is cheating. Everybody is optimising the thing they were asked to optimise, and the dashboard stays green over a system that is getting worse.

This is the same mechanism that drives a model to a shortcut. A model is a thing trying to get a good score, and it will take the shortest path from data to score whether or not you meant it to. A team held to a metric behaves the same way, for the same reason, on a longer timescale.

What to do about it

You cannot escape proxies. You can only stop treating them as the thing.

Write down what each number stands for. One sentence, in plain language, next to the number. This costs nothing and resolves a surprising share of these situations on the spot.

Never report a blended score for independent machines. If two components can fail separately, they get separate numbers. Anything else is a thermostat across two rooms.

Slice before you trust. By source, by type, by age, by customer, by difficulty. Look at the worst slice with real volume, not the mean.

Keep a measurement that is not the target. The moment a number is used to judge people, it starts drifting from reality. Keep at least one measurement that nobody is held to, ideally one with a human in it, and use that as your check on the ones that are.

Let a human read a sample, continuously. One or two answers in a hundred, checked against the source by someone who knows the domain. It is the most expensive layer, the first one cut, and the only place actual truth ever enters the system. Everything else is a proxy checked against another proxy.

Ask what your approach misses. Of any measurement scheme, and of anyone selling you one. A person who really does this has a quick, specific answer, because they live with the limit every day. An answer that it misses nothing is always false.

The rule underneath

Any number you trust is standing in for the real thing, and the gap between the number and reality is exactly where trouble hides, in plain sight.

The 95 percent was not a lie. It was answering a different question from the one everyone thought they were asking, and no amount of watching it more carefully would have revealed that. Only asking what it stood for would.

FAQ

What is Goodhart's law in the context of AI systems? The moment a metric becomes the objective, people and models find ways to move it that are unrelated to what you actually wanted. The metric line keeps climbing while real quality turns and declines, and the dashboard stays green over a system that is degrading.

Why did my AI metric stay green while the system was wrong? Because the metric was standing in for something it was not actually measuring. Common cases: it measures how much your test set resembles your training set, it averages two independent components into one meaningless figure, or it hides a small slice that is failing badly inside a healthy average.

How do I know whether a metric is measuring the right thing? Write down in one sentence what the number stands for, then ask what would have to be true for the number to be high while the real thing is bad. If you can construct that scenario easily, you have found the gap. Also slice the metric, since averages conceal exactly the failures you need.

What should I measure instead of one quality score? Separate numbers for components that can fail separately, sliced by the dimensions your business actually varies along, plus at least one measurement nobody is held to as a target and a continuous small human sample checked against the source.