How to Tell Your Vendor Is Aiming at the Wrong Thing

Mostafa Dhouib·September 6, 2026·7 min read

How to Tell Your Vendor Is Aiming at the Wrong Thing

The short answer. Incompetence is easy to spot and comparatively cheap. The expensive failure is a competent team aimed at the wrong component: the work is good, the updates are detailed, the burn is on schedule, and none of it touches the thing that is broken. Four tells show up before any code does, and all four are visible in a conversation.

Two contractors looked at a speech system that scored 95 percent in testing and 44 percent a week after launch. Both recommended rebuilding the model. One quoted four months, the other six.

The actual fix took five days and the model was never the problem.

Neither contractor was incompetent, and neither was dishonest. They were aiming at the wrong thing, which is a far more common failure than either and considerably harder to detect from the outside, because aiming at the wrong thing produces every visible sign of a healthy project.

Why this is hard to see

When a vendor cannot execute, you find out. Deadlines slip, demos break, the updates get vaguer. The signal is loud.

When a vendor executes well on the wrong component, the signal is inverted. The work is genuinely good. The engineers are genuinely skilled. Progress is genuinely real. The only thing wrong is the target, and the target is the one thing not visible in a status report.

Six months in, the deliverable arrives, it is well built, and the original symptom is unchanged. That is when the organisation concludes the problem is harder than it thought, which is precisely the wrong conclusion.

Tell one: a recommendation before a diagnosis

The clearest signal, and the earliest.

Ask: "which checks did you run, and what did they return?"

A diagnosis produces findings: this artifact hash does not match that one, the input at the model boundary differs from the test input in these two fields, the fallback path was taken this many times yesterday, this source accounts for this share of the training data. Findings are specific and they have numbers attached.

A recommendation without findings is an architecture opinion. It may be a good one. It is not evidence, and it is being priced as though it were.

The tell is the shape of the answer. If you asked what they found and you receive a description of what they would build, no diagnosis happened.

Tell two: one number instead of a breakdown

Ask: "can you split that by phase, with the diagnosis priced separately?"

A single figure for a whole engagement hides which branch it assumes. A failing AI system has four common causes and they differ by more than an order of magnitude in effort, so a quote that does not say which one it assumes is a quote for the most expensive one.

Watch what happens when you ask for the split. A vendor working from evidence separates it easily, because the phases exist in their own head already. A vendor working from a template resists, because splitting it exposes that the diagnosis was never a distinct activity.

Tell three: no answer to what their approach misses

This is the highest-signal question available to you and it is close to unfakeable.

Ask: "what does your approach miss?"

Anyone who has done this work for real has watched their own method fail, and they can tell you the conditions under which it does. "This works well when the failure is in the model and badly when it is in the seams between systems." "We are strong on retrieval quality and we do not do embedded, so if the constraint is on-device we are the wrong people."

Three answers should worry you, in increasing order:

Nothing. Either they have not run their approach into its limits, or they will not tell you. Both are disqualifying for work you cannot easily reverse.

Only things outside their scope. "We do not do hardware" is a boundary, not a limitation. The question is what their method misses inside its own territory.

A repackaged strength. "We are maybe too thorough" is a sales answer. It tells you the conversation is a pitch rather than an assessment.

Tell four: the instruments all point one way

This one requires you to notice an absence, which is harder, so it is worth checking deliberately.

Ask: "what is watching the input distribution and the fallback rate?"

Most organisations have total visibility on the model and near-darkness around everything feeding it. There is a dashboard with latency, accuracy, and version history. Nothing is counting how often a timeout returned a default, or whether today's inputs resemble the training inputs.

A vendor who inherits that asymmetry without remarking on it will investigate where the light is, which is exactly where the problem usually is not. A vendor who notices it and says "before anything else, we need a count of the fallback path" is telling you something useful about how they think.

The question that resolves all four

If you only ask one thing, ask this:

"What is the cheapest experiment that would prove you wrong?"

Anyone confident in a six-month diagnosis should be able to name a small test that would confirm or kill it, usually an afternoon of work. Slice the training data by collector and look at the distribution. Hash the deployed artifact against the evaluated one. Re-split the test set by group and re-measure.

Three responses, and they sort cleanly:

They name one immediately. Good. Ask them to run it first, and make the rest of the engagement contingent on the result.

They say the whole engagement is the experiment. That is a way of saying there is no cheap test, which is rarely true and always convenient.

They are offended. That is the answer.

What to do when the tells are present

You do not need to fire anyone, and an adversarial move here usually costs you more than it saves.

Restructure the sequence instead. Buy the diagnosis as a small, separate piece of work with a finding as its deliverable, from someone with no stake in the build that follows. Then let the build be scoped from the finding. A vendor aimed at the right thing loses nothing under that structure, and will often welcome it because it removes the argument. A vendor aimed at the wrong thing will object to it, and the objection is the information you were looking for.

FAQ

How do I know if my AI consultant is wrong? Look for four tells: a recommendation delivered before any diagnostic findings, a single price with no phase breakdown, no honest answer to what their approach misses, and no attention to the layers that are not instrumented. All four are visible in conversation, before any code is written.

Should I get a second opinion on an AI project? Yes, particularly before approving anything measured in months. Buy the diagnosis separately from someone with no stake in the build that follows, and make the build contingent on what the diagnosis finds. The cost of that is small relative to the spread between the possible fixes.

What is the best question to ask an AI vendor? "What does your approach miss?" It is close to unfakeable. Anyone who has run their method into its limits can describe those limits specifically. Nothing, a list of things outside their scope, or a repackaged strength are all answers in themselves.

Why do good engineers work on the wrong problem? Because the instruments point at the model. There is a dashboard on it with latency, accuracy and version history, and usually nothing watching the input distribution or the fallback rate. Investigation goes where the light is, and the failure is often in the dark.

What if the vendor refuses a separate diagnosis phase? That is informative. A vendor aimed at the right thing loses nothing by having the finding established first, and often prefers it. Resistance usually means the finding was never going to change the recommendation.