Ground Truth

What It Costs to Fix a Failing AI System

Mostafa DhouibMostafa Dhouib··8 min read
The short answer

Nobody in this field answers the cost question straight, so here it is. The price is not set by your system's complexity, it is set by which of four diagnoses you land on, and the four are separated by more than an order of magnitude. Any quote produced before the diagnosis is a quote for the most expensive branch, because that is the only one you can price without looking.

What It Costs to Fix a Failing AI System

The short answer. Cost is not a function of how complicated your system is. It is a function of which of four diagnoses you land on, and those four differ by more than an order of magnitude: hours, days, weeks, months. Nobody can tell you which one you are in without looking, which means any quote arriving before a diagnosis is priced for the most expensive branch. The correct first purchase is the diagnosis, and it should be small enough that getting it wrong costs you almost nothing.

Search for what it costs to fix a failing AI system and you get day rates, engagement tiers, and a contact form. Almost nobody answers the actual question, which is not "what is your rate" but "how much work is this going to be."

That question has a real answer, and the answer has a shape rather than a number.

Why the number is not knowable up front

A failing AI system presents identically across four causes that cost wildly different amounts to fix. The symptom is the same: it worked in testing, it does not work now, and the dashboard says the model is fine.

Plumbingwrong artifact deployed, input mismatch, preprocessing drift, a silent fallback. Hours to days, and the model is not implicated
Dataone source dominates the training set. Rebalance, add real variation, retrain the same architecture. Days
Seamsevery component works and the system does not. Weeks, most of it establishing what is true
Model or task ceilinggenuinely at the limit. Months, and the rarest of the four
The spread between top and bottom is a factor of thirty or more, and no scoping call narrows it. The information that separates the branches does not exist until someone looks.
FigureThe same symptom presents across four causes that differ by more than an order of magnitude in effort. Nobody can tell you which one you are in without looking, which is why the diagnosis is the only thing worth buying first.

Plumbing. The deployed artifact is not the one you evaluated, the input arriving at the model differs from the input you tested with, the preprocessing diverged, or something is timing out and silently returning a default. Hours to days. The model is not implicated at all.

Data. The training set is dominated by one source, so the model learned a fingerprint rather than the task. Rebalance, add the variation the field actually contains, retrain the same architecture. Days.

Seams. Every component works and the system does not. The failure lives between two vendors, two teams, or two layers, and each investigation ends politely at its own boundary. Weeks, and most of that is establishing what is true rather than changing code.

The model or the task. Genuinely at a ceiling. Months, and it is the rarest of the four.

The spread between the top and the bottom is not twenty percent. It is a factor of thirty or more. No amount of scoping calls narrows that, because the information that separates the branches does not exist until someone looks at your data and your traces.

What this means for a quote

Here is the uncomfortable consequence, and it is the reason this page exists.

If someone quotes you before running a diagnosis, they have priced the branch they can see. That is nearly always the bottom of the ladder, because a rebuild is the one option you can specify without knowing anything: new model, new pipeline, six months, here is the plan.

Quoted before looking
Prices the branch you can specify without evidence
That branch is always the rebuild
New model, new pipeline, six months, here is the plan
Fixes everything, which is why it is expensive by construction
Quoted after a diagnosis
Prices the branch the evidence points at
Phased, with the diagnosis costed separately
The build is contingent on the finding
Frequently a rounding error against the alternative
Nobody gets a six-month contract for diagnosing a five-day data fix. That does not make the advice wrong, it makes it advice you should want independently checked.
FigureWhat a quote can contain depends entirely on whether a diagnosis happened first. This is usually not dishonesty, it is what happens when nobody asked what is wrong.

This is usually not dishonesty. It is what happens when the question "what is wrong" has not been asked and the question "what would you do" has. A competent engineer asked to propose a fix with no evidence will propose the thing that fixes everything, and the thing that fixes everything is expensive by construction.

But price the incentive honestly while you are at it: nobody gets a six-month contract for diagnosing a five-day data fix. That does not make the recommendation wrong. It makes it a recommendation you should want independently checked, in the same way you would want a second opinion on any expensive advice from the party being paid for it.

The three tells in a quote

You can read a lot off the shape of the proposal, before you know anything technical.

One number, no breakdownhides which branch it assumesask for it split by phase. A proposal built from evidence splits easily
A rebuild before a diagnosistreatment before diagnosisask which checks were run and what they returned
No answer to what their approach missesclose to unfakeableanyone who has done this work has watched their own method fail and can describe the conditions
FigureYou can read a proposal before you understand anything technical about your own system. Three tells, all visible in the shape of the document.

One number instead of a breakdown. A single figure for the whole engagement hides which branch it assumes. Ask for it split by phase, with the diagnosis priced separately and the build contingent on what the diagnosis finds. A proposal that cannot be split that way was not built from evidence.

A rebuild recommended before a diagnosis. Ask which checks were run and what they returned. If the answer is a description of your architecture rather than a list of findings, the diagnosis has not happened.

No answer to "what does your approach miss." This is the single most informative question you can ask any vendor, and it is close to unfakeable. Someone who has done this work has watched their own approach fail and can tell you the conditions under which it does. Someone who says nothing, or lists only things outside their scope, is telling you they have not.

How to buy it instead

The structure that protects you is not a better rate. It is a different sequence.

Buy the diagnosis first, separately, and small. It should be days rather than months, priced so that being wrong about the vendor costs you a rounding error rather than a quarter. Its deliverable is a finding with evidence, not a proposal.

Make the build contingent on what the diagnosis says. Write it into the engagement: the fix is scoped after the finding, from the finding. Anyone unwilling to work that way is telling you the finding was never going to change their answer.

Require the cheap-experiment answer. Ask what the cheapest test would be that proves the diagnosis right or wrong. There is always one, and it is usually an afternoon. Confidence that cannot name such a test is not coming from evidence.

Converting the ladder into your own numbers

Since I am not going to invent a rate card, here is how to build your own estimate in about ten minutes.

Take your fully loaded cost for one senior engineer-week, internal or contracted, whichever you would actually use. Then:

BranchEffortMultiplier on one engineer-week
PlumbingAn afternoon to two daysUnder 0.5
DataThree to ten days1 to 2
SeamsTwo to six weeks2 to 6
Model or task ceilingTwo to six months8 to 25

The diagnosis that tells you which row you are in sits below all of them, and it is the only line item you should be willing to buy without knowing the answer to anything.

The reason to write it out this way is that it reframes the decision. You are not choosing between vendors on price. You are deciding how much you are willing to spend to find out which row you are in, and the answer to that should be: much less than the difference between the rows.

The failure that costs more than the money

One more cost belongs on the ladder and it never appears in a proposal.

A rebuild typically retrains on the same data through the same pipeline. If the cause was in the data, the new model learns the same shortcut and lands on the same number. At that point the organisation does not conclude that it made the same mistake twice. It concludes that AI does not work for this problem and walks away from something that would have worked.

That is the expensive outcome, and it is not measured in the engagement. It is measured in the capability you no longer believe you can have.

FAQ

How much does it cost to fix a failing AI model? It depends entirely on which of four causes you have, and they differ by more than an order of magnitude: plumbing takes hours to days, a data-distribution problem takes days, a seam failure takes weeks, and a genuine model ceiling takes months. The diagnosis that tells you which one is a small, separate purchase, and it should come before any quote.

Why won't anyone give me a price for AI consulting? Because the honest answer requires looking first, and the incentive runs the other way. A vendor asked to quote without evidence will price the branch that fixes everything, which is the most expensive one. Ask for the diagnosis priced separately and the build scoped afterwards from what it finds.

What should an AI diagnosis cost? Small enough that being wrong about the vendor is a rounding error rather than a quarter. Days rather than months, with a deliverable that is a finding backed by evidence rather than a proposal for further work.

How do I know if a quote is aimed at the wrong thing? Three tells: one number instead of a phased breakdown, a rebuild recommended before any checks were run, and no answer to what their approach misses. The last one is close to unfakeable, because anyone who has done this work has watched their own method fail.

Is a rebuild ever the right answer? Yes, but it is the rarest of the four branches and it should be the conclusion of a diagnosis rather than its substitute. A rebuild on the same data through the same pipeline reproduces the original failure, and the second failure usually costs the organisation its belief that the problem is solvable at all.

Free worksheet
The Pre-Rebuild Diagnostic

The four boring checks and the data-distribution slice, ending in a rebuild-or-repair verdict with the evidence for it. An afternoon of work, and it is designed to be carried into the meeting where somebody is proposing six months.

One email, the resource, and nothing else unless you reply.
Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program