The Model Is Five Percent: Where Production AI Actually Fails

Mostafa Dhouib·September 4, 2026·10 min read

The Model Is Five Percent: Where Production AI Actually Fails

The short answer. The model is roughly five percent of a production AI system. The other ninety-five percent is the pipelines, the protocols, the drivers, the firmware, the silicon, and the seams between them, and that is where these systems actually fail. The strongest evidence is not an argument about AI at all: the identical failure pattern appears, with the same causes and the same fixes, in large systems that contain no model anywhere.

Every engagement that begins with "the model is broken" has the same structure. The output is wrong, the model produces the output, so the model is the problem. It is a clean syllogism and it is almost always false.

This is the thesis the rest of the writing on this site is evidence for, so it is worth stating directly and then testing it as hard as I can.

The argument

A production AI system is not a model with some engineering around it. It is a large, layered system that happens, sometimes, to contain a model.

Count the layers honestly. There is the data collection and everything that shapes it. The ingestion and preprocessing. The feature and retrieval path. The model. The orchestration around the model. The verification, or its absence. The state and the handoffs between steps. The protocols carrying the data. The drivers and firmware underneath those. The silicon those run on. And a deployment, monitoring, and operations layer wrapped around the whole thing.

The model is one box. It is the box with a name, a version, a benchmark, and a dashboard, which is exactly why it absorbs the attention.

Four pieces of evidence

The claim is falsifiable, so here is what actually supports it.

The arithmetic makes it inevitable

Reliability multiplies across steps. At 95 percent per step, a five-step chain delivers 77 percent. A twenty-step task delivers about 36 percent, and a forty-step task about 4 percent, with the model and the prompt unchanged across all of them. The only variable is how many steps.

That has a consequence people rarely follow through. If you want a twenty-step system to work 95 percent of the time end to end, each step needs 99.7 percent, not 95. The gap between "a good model" and "a working system" is two orders of magnitude on the error rate, and no model release closes it.

The reliability you lose does not live inside any single step. Audit each one and they all come back green. It lives in the connections, which is exactly the region no component owner is responsible for.

The research agrees

Researchers categorised failures across hundreds of multi-agent execution traces. About 40 percent came from bad specification and design, about 37 percent from coordination and handoffs, and about 21 percent from nobody verifying the work.

Specification, coordination, verification. That is the whole pie, and none of it is the model.

The war stories keep landing outside the model

A speech system collapsed from 95 percent to 44 percent in production. The cause was one person having recorded nearly all the training data, and the fix was five days of data work against a proposed four-to-six-month rebuild of a model that was fine.

A retrieval system was quietly wrong on scanned documents at 41 percent while the average read 88. The fix was slicing a number, not rebuilding a model.

An agent chain inherited one missed document and produced confident nonsense five steps later. The fix was letting each step reject a bad input, which cut the cascade by 82 percent. Same model, same prompts.

An agent reported success while zero tool calls executed. The call was correct every time and was dropped in the plumbing between the model and the tool.

An agent ran green for 264 hours and billed 47,000 dollars, with every individual call correctly bounded and nothing bounding the system of loops.

A biosignal front end went flat whenever the actuator fired. The cause was the return current coming home through the reference electrode, and the fix was a decision about where current returns.

Not one of those was fixed by changing a model.

The controlled experiment: systems with no model at all

This is the piece that convinces me, because it removes AI from the equation entirely.

The same disease appears in a fleet of 25,000 servers coordinating over hardware protocols, in telemetry pipelines moving data off physical machines, and in protocol integration reaching down to the silicon. No model anywhere in any of them. Same compounding small errors, same invisible failures, same coordination faults, same absence of verification between stages.

If the pattern were about AI, it would not appear in systems with no AI. It appears there in exactly the same shape, which means the pattern was never about AI. It is about production systems and the space between their parts.

The strongest objection

The fair pushback: is this just a claim that everything is systems engineering, which is unfalsifiable and therefore useless?

No, and the difference is that this makes specific predictions that could fail.

It predicts that swapping in a better model will not fix these systems. That is testable, and it is tested constantly, because it is the first thing every team tries. The result is consistent: a stronger model in an unchecked loop produces more articulate, more confident, more expensive failures. You do not get fewer errors, you get better-dressed ones.

It predicts that per-component monitoring will show green while the system fails, because the failure is in the composition. That is observable, and the planner and executor resetting each other's progress measure is a case where both dashboard panels are correct and the system never terminates.

It predicts that the fixes will be unglamorous and cheap relative to the alternative. Five days against six months. An idempotency key. A check comparing intended against executed. A calendar.

And there is a case where the thesis is simply wrong, which is worth naming: sometimes the model genuinely is the problem. When the task is at the edge of what the architecture can represent, when the data is adequate and correctly distributed and the ceiling is real, the model is the constraint. That case exists. It is rarer than the reflex assumes, and the four boring checks plus a look at the data distribution will tell you which one you are in within an afternoon.

Why the attention lands on the wrong box

None of this is a competence problem, and it is worth being precise about the mechanism rather than treating it as carelessness.

The model is visible. It has an architecture you can draw, a version you can bump, a benchmark you can compare, and a vocabulary the whole team shares. The data sits in a folder. The preprocessing is a module nobody wants to open. The handoffs between steps are not owned by anyone in particular.

Then look at where the instruments point. There is a dashboard on the model showing latency, accuracy, version history, a whole toolchain watching the one component that was fine. Ask what is watching the input distribution, the fallback rate, the intended-versus-executed count, or the aggregate spend per task. Usually nothing.

Total visibility pointed at the healthy layer, total darkness around the broken one. That is not a people problem. It is a where-the-instruments-point problem, and it is fixable in a week.

There is a commercial gradient underneath it too, and it deserves saying plainly. Rebuilding feels decisive. "Fix your data" sounds too small to be the real answer for a system that matters, and it quietly points at whoever collected the data. And nobody gets a six-month contract for telling you it is a five-day fix. The people recommending the rebuild are usually not being dishonest. The expensive answer simply happens to be the one that pays.

What follows for how you staff it

If the risk concentrates in the seams, a firm that owns one layer cannot own the risk, however good it is at that layer. It can do excellent work up to the edge of its scope and hand the problem across.

That is not a criticism of specialists. It is a structural observation: the failure almost always lives in a layer nobody thought to open, and a deeper specialist just has a taller lane. You do not fix a seam by hiring more depth. You fix it by making someone accountable for the span.

What to do on Monday

The practical version of the thesis is short.

Instrument the ninety-five percent. Every step recorded, inputs and outputs, a trace identifier through the whole task, and a count of intended against executed at every boundary. Verify between steps with something that is not the model grading itself. Let a step reject a bad input rather than doing its best with garbage. Bound the system, not the loop, and sort your actions into reversible and irreversible before you gate anything.

Then run the arithmetic on your own system. Count the steps, take an honest per-step reliability, multiply, and compare it against what you are promising. That calculation takes ten minutes and it has ended more architecture debates for me than any other single thing.

If the number comes out where it usually does, the model was never your problem.

The vocabulary this argument uses, the seam, load-bearing wrong, the third state, and the rest, is defined in the glossary. Every figure quoted above is listed with its context on the numbers.

FAQ

Why do production AI systems fail if the model is good? Because reliability multiplies across steps and the losses live in the connections, not inside any component. At 95 percent per step a twenty-step task finishes about 36 percent of the time with every step passing its own test. Research across hundreds of traces attributes roughly 40 percent of failures to specification, 37 percent to coordination, and 21 percent to missing verification.

Is the model ever actually the problem? Yes, and it is worth naming: when the data is adequate and correctly distributed and the task sits at the edge of what the architecture can represent, the model is the constraint. That case is real and much rarer than the reflex assumes. Four plumbing checks and a look at the data distribution will tell you which case you are in within an afternoon.

How do I know the failure is in the system and not the model? Confirm the deployed artifact matches the evaluated one, the inputs match, the preprocessing is numerically identical, and nothing is silently falling back. Then slice the training data by source and the production score by segment. If all of that is clean, you have earned the right to suspect the model.

Why does swapping in a better model make things worse? Because a stronger model in a loop with no verification argues more convincingly from the same wrong starting point. You get more articulate, more confident, and more expensive failures rather than fewer, and the improved fluency makes them harder to spot in review.

Does this apply to systems that do not use AI? That is the strongest evidence for it. The same compounding errors, invisible failures, coordination faults, and missing verification appear in a 25,000-server fleet coordinating over hardware protocols with no model anywhere. The pattern was never about AI.