You Can't Fix What You Can't See: Instrumenting an AI Agent

Mostafa Dhouib·September 4, 2026·7 min read

You Can't Fix What You Can't See: Instrumenting an AI Agent

The short answer. A system with 11 internal steps logged one bit: did the whole thing work. All 11 steps in between were dark, so when it failed it looked random, and a good team told me it fails randomly and cannot be reproduced. There is no such thing as random failure. There is only failure you cannot see, and the randomness is not in the system, it is in your visibility. Turning the lights on took two days and showed 85 percent of failures coming from one specific step, which then took two days to fix.

I hear one sentence more than any other, and I have stopped being polite about it: it just fails randomly, we cannot reproduce it.

The teams saying it are not bad teams. They are usually good ones, running profitable businesses, with real engineering discipline everywhere else. But the sentence is not a description of the system. It is a description of what they can see.

The 11-step black box

The system had 11 distinct internal stages of reasoning and action. It logged exactly one thing at the end: success or failure.

One bit, for 11 stages. Everything in between was dark.

Consider what that makes debugging. A run fails. You know it failed. You do not know which stage, what it received, what it produced, what it decided, or whether it even reached the stage you suspect. Two failures that look identical from the outside can have entirely different causes, and there is no way to group them.

Under those conditions failures genuinely do look random, because the only signal you have is a coin flip with no covariates. The pattern is there. You have no instrument that can resolve it.

What turning the lights on actually found

We did the obvious thing nobody had done, and instrumented every step.

It took two days. Then 85 percent of the failures resolved to one specific stage out of the 11. Once the team could see it, the fix took another two days.

The moral is not that the team was careless. It is that this was never an unreliable system. It was one broken piece hiding in plain sight, because nobody was watching the middle. The second the middle became visible, the mystery evaporated and stopped being a mystery at all.

That ratio is typical in my experience. When a system that appears broadly unreliable becomes observable, the failures usually concentrate hard rather than spreading evenly. Broad unreliability is what a concentrated failure looks like through a one-bit instrument.

This is not a new problem

If this sounds like an AI-specific issue, it is worth zooming out, because the pattern is much older and the stakes have been much higher.

In the 2003 blackout that cut power to roughly 50 million people in the northeast, part of the reason it got as bad as it did was a software fault that quietly stalled the alarm system in the control room. The operators sat there flying blind while the grid cascaded around them.

It was not the failure that did the damage. It was the view of the failure going dark. And the operators trusted the quiet, because quiet is supposed to mean everything is fine.

That is the same disease as your agent. A small failure compounding through a system nobody can see, while every available signal reports normality. Larger blast radius, identical mechanism.

What to actually record

"Add observability" is not an instruction. Here is the specific list, roughly in order of value per hour spent.

A trace identifier through the whole task. One identifier carried through every step and every nested call, so a tree of agent runs is recognisable as a single task rather than four hundred unrelated ones. Without this, nothing else you record can be joined, and it is the reason a runaway call tree stays invisible.

Per-step input and output. What the step received and what it produced. This is what makes a failure reproducible, and reproducibility is the entire difference between a bug you can fix and a mystery you can only describe.

The decision at each branch. Not just what happened, but which way the system chose and on what basis. Most agent failures are a wrong branch taken confidently, and without the branch record you see the consequence and never the choice.

Intended against executed. At every boundary where a model asks for something and code does it, count both numbers. One emitted, zero executed is an entire class of failure expressed as a pair of integers, and nothing else in your system will surface it.

Cost and latency per step, and per task. Aggregated at the task level, not just the call level. A per-call dashboard makes a runaway look healthy, because every individual call was fine.

A distinguishable terminal state. Done, gave up, or killed, at the return type rather than in a log line. If a run that hit its cap and a run that finished look the same to the caller, your instrumentation is recording the wrong thing at the most important moment.

The test for whether you have it

There is one question that settles whether your instrumentation is real, and it is not about coverage percentages.

When a run fails, can you name the step that broke it in under ten minutes, without reproducing it?

If yes, you have observability. If no, you have logs. The distinction is not pedantic: logs tell you what happened at points somebody thought to write about, and observability lets you ask a question nobody anticipated. The failures that matter are always the ones nobody anticipated.

A second, sharper test: take a failure from last week and try to answer why it failed using only what was recorded at the time. Not by rerunning it, because reruns of nondeterministic systems answer a different question.

Why this is the first thing to build

Everything else in agent reliability is guesswork without it.

You cannot verify between steps if you cannot see the steps. You cannot know which of the four doors a duplicate came through without identifiers and timings. You cannot tell a dropped tool call from a model that never asked without both numbers. You cannot bound a call tree you cannot recognise as one task. You cannot even tell whether your fix worked.

It is also the cheapest item on the list, and the only one whose value does not depend on being right about the diagnosis first.

So when a team asks me what to build first, the answer is always the same, and it is never the interesting answer: turn the lights on. Then the system will tell you what is wrong with it, usually within a couple of days, and usually it is one step.

FAQ

Why does my AI agent fail randomly? It does not. You cannot see it fail. A system that logs only a final pass or fail gives you a coin flip with no covariates, so real patterns are unresolvable. Instrument every step and the randomness usually collapses to one stage causing most of the failures.

What should I log in an AI agent system? A trace identifier carried through every nested call, per-step inputs and outputs, the decision taken at each branch, intended against executed counts at every model-to-code boundary, cost and latency aggregated per task as well as per call, and a terminal state distinguishable at the return type.

How do I know whether I have real observability? Ask whether you can name the step that broke a failed run in under ten minutes without reproducing it. If yes, you have observability. If no, you have logs. Then try answering why a failure from last week happened using only what was recorded at the time.

What should I build first to make an agent reliable? Observability, because every other decision is guesswork without it and it is the cheapest item available. You cannot verify between steps you cannot see, diagnose duplicates without identifiers and timings, or bound a call tree you cannot recognise as a single task.

How long does instrumenting an agent take? In the case described here, two days to instrument 11 steps and two more to fix what it revealed. The work is unglamorous and mechanical, which is the main reason it gets deferred in favour of investigations that cost far more.