Rebuild or Repair? How to Tell Before You Approve Six Months

Mostafa Dhouib·September 5, 2026·8 min read

Rebuild or Repair? How to Tell Before You Approve Six Months

The short answer. A rebuild is the most expensive option available and it is frequently aimed at the component that was working. Worse, a rebuild usually trains on the same data through the same pipeline, so it reproduces the same failure and the organisation concludes that the technology does not work for them. The choice between rebuilding and repairing is not a judgement call. It is a diagnosis that takes an afternoon, and it has a decision table at the end of it.

A speech system scored 95 percent in testing and 44 percent a week after launch. Two contractors had already looked at it. Both said rebuild the model; one quoted four months, the other six.

The actual fix took five days, and the model was never the problem.

Neither contractor was incompetent or dishonest. They were aiming at the wrong thing, which is a far more common failure than either, and considerably harder to detect from the outside. This article is how to detect it before you sign.

Why the default answer is a rebuild

Before the procedure, the reason you need one. Four forces push toward rebuilding, and none of them is evidence.

A rebuild feels decisive. It is a plan, a budget, a timeline, and a visible response to a serious problem. "Go and look at your data" sounds too small to be the real answer for a system that matters.

The cheap answer points at someone. If the cause is the training data, that quietly implicates whoever collected it. Nobody means it that way, and everybody feels it. The rebuild has no name attached, which makes it the socially safe recommendation.

The instruments point at the model. There is a dashboard on it, with latency, accuracy, and version history. Ask what is watching the input distribution or the fallback rate and the answer is usually nothing. Total visibility on the healthy layer, darkness around the broken one.

And the expensive answer pays better. Nobody gets a six-month contract for telling you it is a five-day data fix. The people recommending a rebuild are usually not being dishonest, but the incentive runs one way, and you should price that in the way you would with any other advice you did not pay for independently.

The diagnosis, in three parts

Run these in order. The order is most of the value, because each part is cheaper than the one after it.

Part A: the four plumbing checks

The model is not the first suspect. It is not usually in the first four. Every one of these is an afternoon at most, and most reported model failures die here.

Is the same model actually running? Hash the artifact serving production and the artifact you evaluated, and compare the bytes rather than the version string. Stale deploy paths, moved registry tags, and temporary fallbacks still serving traffic are all common.

Is it getting the same input? Capture a real production input at the point the model receives it and compare against a test input at the same point: shape, dtype, range, sample rate, channel order, units.

Is the preprocessing numerically identical? Run one raw input through both pipelines and diff the tensor immediately before the model. Numerically, not by eye. Two inputs that look identical to a person can differ enough to change the model's behaviour.

Is anything timing out or falling back? Count how often the fallback path was taken yesterday. These return defaults without raising, so the model looks bad when it was handed a zero or never ran. If nobody is counting, that is itself the finding.

Part B: the data distribution

If all four pass, that is a real result rather than a wasted afternoon. It means the system did not break: it generalised badly, and it was always like this.

Slice the training data by every axis you have. Who collected it, which device, which site, which source, which time window. You are looking for one bar that dwarfs the others.

In the 95-to-44 case, one person, the engineer who built the first version, had recorded nearly all the training data. The model never learned the commands. It learned him: his voice, his microphone, the room he recorded in. Every one of those was a fingerprint sitting next to the correct answer thousands of times, and the model took the shortest path from data to score, as models do.

The moment that chart exists, 44 percent stops being surprising. It is the number you would predict.

Part C: how the test set was split

One more check, and it explains the 95 percent rather than the 44.

If the test set was split at random, the dominant source landed on both sides. The model was asked to recognise the thing it was built around, on data from the same room and the same equipment. Its shortcut works perfectly on that test.

A random split is structurally incapable of detecting a shortcut keyed on a group that appears on both sides. Not too small, not badly chosen: incapable. Adding more test data drawn the same way would change nothing.

Split by group instead, holding out whole speakers or sites or devices, and this system would have reported something near 44 percent on day one, before anyone proposed a rebuild.

The decision table

Only the last row justifies a rebuild, and in my experience it is the rarest row in the table.

What the repair looked like

For the record, because "fix the data" is vague enough to sound like hand-waving.

Balance the dataset so one voice does not dominate. Add the real-world noise the field actually contains rather than the clean conditions the original recordings had. Retrain the same model, unchanged architecture. Re-measure with a group split so the number means something.

Five days. Back above 99 percent. No rebuild, no six months, no six figures.

The part that costs more than the money

The rebuild in that story was approved before I arrived, and the outcome is worth stating because it is the real risk.

Six months, the whole team, training a new model on the same data through the same pipeline, because nobody suspected the data. The new model learned the identical shortcut and collapsed to the same number.

At that point the organisation does not conclude that it made the same data mistake twice. It concludes that AI does not work for this problem, and walks away from something that would have worked. The failure resells itself, and the second sale is far more expensive than the first, because it costs the capability rather than the quarter.

Five questions for the meeting

Which of the four plumbing checks did you run, and what did they return?

Can you show me the training data sliced by collector, device, site, and time period?

Was the test set split at random or by group, and which group?

If we rebuild on the same data through the same pipeline, what specifically stops the same failure?

What is the cheapest experiment that would prove you right? Anyone confident in a six-month diagnosis should be able to name a cheap test that would confirm it. If there is no such test, the confidence is not coming from evidence.

FAQ

Should we rebuild our AI model or fix it? Run the diagnosis before deciding. Check that the deployed artifact matches the evaluated one, that inputs and preprocessing are identical, and that nothing is silently falling back. Then slice the training data by source and check whether the test set was split at random or by group. Only if all of that is clean is a rebuild the right call.

Why do consultants recommend rebuilding a model? Usually because a rebuild feels decisive, the cheap answer implicitly blames whoever collected the data, and every instrument in the organisation points at the model rather than at the data. There is also an incentive: nobody gets a six-month contract for diagnosing a five-day fix. That is a reason to run the diagnosis yourself rather than to assume bad faith.

How long should diagnosing a failing AI system take? An afternoon for the plumbing checks, a day or two for the data distribution and split analysis. If someone proposes months of work before producing that analysis, they are proposing treatment before diagnosis.

Why does rebuilding often reproduce the same failure? Because a rebuild typically retrains on the same data through the same pipeline. If the cause was a shortcut learned from an unbalanced dataset, the new model learns the identical shortcut. The organisation then concludes the technology does not work, rather than that the same mistake was made twice.

What did the actual five-day fix involve? Balancing the dataset so one source no longer dominated, adding the real-world noise the field contains, retraining the same unchanged model, and re-measuring with a group split so the number meant something. Accuracy went back above 99 percent.