Why Your AI Works in Testing but Fails in Production

Mostafa Dhouib·15 min read

Why Your AI Works in Testing but Fails in Production

The short answer. A model that scores 95% in staging and 44% in production a week after launch has almost never broken. It learned a shortcut in your training data that predicts the right answer in the room it was trained in, and that shortcut disappears the moment real users arrive. The common fix, rebuild the model, retrains on the same data and reproduces the same collapse. The real fix is usually the data, and usually days, not months.

A founder called me about a speech to command system running on a small edge device. In staging it tested at 95%. A week after launch, with real customers, it was sitting at 44%. More than half wrong, overnight. That is not a clean number on a dashboard, that is a wall of support tickets and customers saying the product does not understand them, and a founder staring at the end of the company.

Two contractors had already looked at it before me. Both recommended the same thing: rebuild the model. One quoted four months, the other six. The actual fix took five days, and the model was never the problem. I have seen this exact pattern across defense, industrial automation, fintech, healthcare, and more than nine other industries, where getting it wrong ran from six figures to millions. It is the same shape every time, and the interesting question is not why two smart people got the diagnosis wrong. It is why the cause was sitting in plain sight and everyone walked straight past it.

The 95% was never measuring what you think

Start with what a test score actually is. A high accuracy number in staging does not measure whether the system works. It measures how much your test set looks like your training set. If those two overlap, the score is high and it tells you almost nothing about the field.

That is the trap. The number that is supposed to help you find the truth becomes the thing that hides it, because once everyone sees 95% they trust the system and stop digging. The score becomes the reassurance, and the reassurance is what keeps anyone from looking at the layer where the problem actually lives.

First, rule out the boring stuff

Before I touch anything interesting, I check the plumbing, the same way a doctor rules out the common causes before the exotic ones. The model is not even my first suspect. Four boring questions:

• Is the exact same model running in production as the one that was tested? You would be surprised how often it is not.
• Is it getting the same input? Same audio format, same sample rate.
• Is the preprocessing identical? If production cleans the input even slightly differently, the model is effectively hearing a different language.
• Is anything quietly timing out or falling back to a default?

Most of the time, when a founder tells me the model is broken, the issue dies right here, in something dumb in the plumbing. In this case it was all clean: same model, same inputs, same pipeline. And that tells you something important. The model did not break. It generalized badly. It learned something that worked in the room it was trained in and stopped working the moment it left that room. Which points straight at the layer nobody watches: the data.

The model did not break, it learned the wrong thing

I pulled up the training data. At first glance it looked fine, tens of thousands of samples, every command, every kind of environment. But raw totals tell you nothing. The real question is never how much data you have, it is how the data is distributed inside itself.

So I sliced it, by command, by environment, by device, and by who recorded it. And there it was. Broken down by speaker, one bar covered almost the entire dataset. One person, the engineer who built the first version, had recorded nearly all of the training data. The moment you see that chart, 44% is not surprising anymore. It is the number you would expect.

The model never learned the commands. It learned him. His voice, his pronunciation, his cadence, his specific microphone, the hum of the room he recorded in. Every one of those was a little fingerprint sitting right next to the correct answer, thousands of times. The model did the most natural thing in the world, it latched onto the fingerprint instead of the words.

Why a model takes the shortcut

Strip a model back to what it actually is. A machine learning model is a thing trying to get a good score. That is it. It can learn the actual task, which is hard, or it can grab whatever signal happens to predict the right answer, which is easy. It always takes the easy path, not because it is broken, but because shortcuts work. If one thing in your data lines up with the right answer often enough, the model leans on it, whether you meant it to or not.

The model has no concept of what you want. It does not know there is a task, it does not know you care about the commands rather than the microphone. All it has is data on one side and a score telling it how it is doing on the other, and it finds the shortest path between the two. It optimizes the data you gave it, not the thing you meant to build. Build the dataset out of one person, and one person is what it learns.

Then real customers showed up. For the first time, the system was meeting strangers instead of the person it was trained on. It fell over, but not for any of the reasons everyone reached for. Not the architecture, not a too-small model, not a missing framework. It had learned the wrong lesson from the data.

Why the test set could not possibly have caught it

There is a mechanical reason the 95 percent was meaningless, and it is worth understanding because it is the most preventable part of the whole story.

The test set was split off the training data at random.

Think about what that does when one speaker dominates the dataset. A random split scatters that speaker's recordings across both sides of the line. He is in the training set, thousands of times, and he is also in the test set. So the test is asking the model to recognize the voice it was built around, on data collected in the same room, on the same microphone, on the same day.

The model's shortcut works perfectly on that test. Of course it scores 95 percent. The test set and the training set are the same distribution because they came from the same recordings, and a random split cannot separate what a group split would have.

This is the part that makes the failure structural rather than unlucky. A random split is incapable of detecting a group-level shortcut. It is not that the test was weak, or too small, or badly chosen. It is that the split strategy guaranteed the shortcut would survive the test, and no amount of adding more test data drawn the same way would have changed it.

The fix is to split by the grouping variable rather than by the sample. Hold out entire speakers, entire sites, entire devices, entire time periods, entire customers. Then the test measures whether the model generalizes to a new one of those, which is the question production is actually going to ask.

Split by group and this system would have reported something close to 44 percent on day one, before the rebuild was ever proposed. Same data, same model, honest number.

How to find out what your model actually learned

If you want to check this on your own system, there are four moves, and none of them requires a research background.

Slice the training data by every metadata axis you have. Who collected it, which device, which site, which source system, which time window. You are looking for a bar that dwarfs the others. This is the chart that made 44 percent stop being surprising, and it took minutes to produce.

Re-split by group and re-evaluate. Hold out whole groups rather than random samples and rerun your existing evaluation. No retraining required for the first look. If the number falls apart, you have your answer, and you got it in an afternoon.

Ablate the suspected shortcut. If you think the model is keying on something incidental, remove or neutralize it and see what happens. Performance that collapses when you strip a feature that should be irrelevant tells you exactly what the model was leaning on.

Test on a source that appears nowhere in training. One new speaker, one new site, one new device. A handful of samples from a genuinely unseen group is worth more than thousands of samples from a seen one, because it is the only measurement that answers the production question.

The reason these are not standard practice is not difficulty. It is that all four require opening the data, and none of them require touching the model, which makes them feel like they are not real work.

This is not a voice problem, it is a shape

Once you have seen this enough times, you stop seeing it as an AI problem and start seeing it as a shape that shows up everywhere:

• An industrial control system tuned perfectly in the workshop that started making bad calls in the field. It had learned the workshop, not the job.
• A famous image model everyone thought was recognizing an animal. It was detecting the snow in the background.
• A fraud model that learned which data source a transaction came from instead of whether it was fraud.

Different industries, different teams, different systems, the same mistake underneath all of them. The AI learned the room it was trained in, and everyone mistook that for the AI understanding what they wanted.

Why smart people walk right past it

These are not dumb people. Usually several smart people miss it at once, and the reason has little to do with intelligence. It is that the model is the part you can see, the code, the architecture, the thing you can open up and argue about. The data just sits in a folder. Nobody gets excited about opening a CSV or analyzing a set of voice recordings, so everyone crowds around the visible part and the real cause sits in the dark.

Look at where the team's attention points. They have dashboards on the model, latency, accuracy, version history, a whole toolchain watching the one thing that was fine. And what is watching the data? Almost always nothing. It gets loaded once and never looked at again. Total visibility pointed at the healthy layer, total darkness around the broken one. That is not a people problem, it is a where-the-instruments-point problem.

The expensive trap: the rebuild reproduces the failure

Here is why this gets expensive. The rebuild got approved. Six months, the whole team, to rebuild a model that was never the problem, and they trained the new model on the same data, because nobody suspected the data. So the new model learned the exact same lesson, took the same shortcut, and collapsed to 44% again. The failure quietly resells itself.

And the money is not the worst part. The rebuild fails, and the company concludes not that they made the same data mistake twice, but that AI does not work for them, and walks away from something that would have worked perfectly if one person had looked at the right chart. That is the real cost. The failure does not just burn six figures and six months. It convinces smart people that something possible is impossible.

So why does the rebuild keep getting recommended? Less competence, more psychology. Rebuilding feels like progress, it feels decisive. Saying "go fix the data" sounds too small, too cheap to be the real problem for a system that matters. The data answer also quietly points a finger at whoever collected it, so the safest option is the rebuild, which has no name attached to it. And the part nobody says out loud: nobody gets a six-month contract for telling you it is a five-day data fix. The people recommending a rebuild are usually not being dishonest, but the expensive answer happens to be the one that pays them the most.

The one habit that prevents all of this

The rule underneath all of it: any number you trust is standing in for the real thing. A test score, a KPI, a demo. The gap between the number and reality is exactly where trouble hides, in plain sight. The 95% was not a lie. It was answering a different question than anyone thought they were asking.

So, two habits, and one of them is the whole game.

When you build something, do not let your own team be the test. Hold real users out, different voices, different devices, different rooms, and map that to whatever you are building. If it only works on the people who built it, you want to find that out on a quiet Tuesday, not from customers in week one. And watch the inputs, not just the scores. The score tells you how the model is doing. The inputs tell you whether the real world still looks like your training data, because the day your real users stop looking like your training set is the day the score becomes a liar, and that day always comes.

The one habit: before anyone puts a number or a timeline on a fix, find out what the model actually learned. Not what you meant it to learn, what it picked up from the data you gave it. In practice that is a few boring questions. Who made your data, and does it trace back to one person or one source? Does your test set look anything like your real users, or is your team quietly testing itself? What changed between the demo and real life? That gap is where the story lives.

Once we knew what this model had actually learned, the fix was simple. We balanced the data so it was not all one voice, added real-world noise, retrained the same model, and it went back over 99%. Five days. No rebuild, no six months, no six figures. The model was never broken.

What stuck with me was not that the fix was clever, it was not, it was simple. It was how close the company came to spending six months solving the wrong problem, when the only thing standing between them and the right answer was a willingness to look at the boring things first. You do not need a machine learning PhD to catch this. You need someone willing to open the folder nobody wants to open.

If you have something that worked in testing and fell apart the moment real people touched it, tell me what is going on and I will tell you where I would look first.

FAQ

Why did my model score 95% in testing and fail on real users? Most often because the test set was split off the training data at random, so the source that dominates your training data also appears in your test set. The model's shortcut works perfectly on that test. A random split is structurally incapable of detecting a group-level shortcut, no matter how much test data you add.

How should I split train and test data to catch this? Split by the grouping variable rather than by the sample: hold out entire speakers, sites, devices, customers, or time periods. Then the test measures generalization to a new one of those, which is the question production will ask.

Why does an AI model pass every test in staging and fail in production? Because a staging score measures how much your test set resembles your training set, not whether the system works. If both were drawn from the same narrow source, the score is high and misleading. In production the inputs stop resembling the training data and the model collapses.

Should I rebuild the model when accuracy drops in production? Almost never as the first move. A rebuild trained on the same data reproduces the same failure, because the model relearns the same shortcut. Find out what the model actually learned from the data before committing to a rebuild. The real cause is usually in the data and fixable in days.

How do I tell whether the problem is the model or the data? Rule out the plumbing first, same model, same inputs, same preprocessing, nothing timing out. If that is clean, slice the training data by source, device, environment, and who recorded it. A single dominant source means the model learned that source's fingerprint, not the task.

What is training-serving skew? It is the gap between the conditions a model was trained and tested in and the conditions it meets in production. When that gap is large, offline metrics stay high while real-world performance drops, because the model optimized for the training distribution rather than the task.