Your Green Dashboard Is Not Evidence

Mostafa Dhouib·September 6, 2026·7 min read

Your Green Dashboard Is Not Evidence

The short answer. There are two dials and most organisations have built one. Monitoring tells you the system is running: latency, error rate, uptime, throughput. Evaluation tells you the system is right: whether the outputs are correct, on which slices, and whether that has changed. A system can be perfectly healthy on the first dial while producing wrong answers on every request, and it will not turn a single indicator amber while it does.

After an incident, the sentence everyone reaches for is "why didn't monitoring catch it?"

Usually the honest answer is that monitoring caught everything it was built to catch. It was built to watch whether the system was running. The system was running. It was running and producing wrong answers, and there was never an instrument pointed at that.

Two dials

The running dial is what you already have. Latency percentiles, error rates, request volume, queue depth, memory, uptime, deploy status. It is mature, it is well tooled, and every engineer knows how to read it. It answers: is the machine turning?

The right dial is usually missing. Accuracy on a maintained evaluation set, broken out by slice. Retrieval recall. The rate at which the fallback path is taken. The share of runs that ended by exhausting a budget rather than by finishing. Input distribution drift against the training distribution. It answers: is the output correct?

The critical property is that these two are independent. Not loosely coupled, not usually correlated. Independent. A wrong answer costs the same latency as a right one, returns the same HTTP 200, and consumes the same memory. Nothing about being wrong is visible on the running dial, because wrongness is not a resource.

What a green dashboard looks like during a failure

Make it concrete, because the abstraction is easy to nod at and hard to act on.

A speech system scores 95 percent in testing and 44 percent a week after launch. During that week: latency normal, error rate zero, uptime clean, throughput at forecast. Every indicator green, every indicator honest. The system was doing exactly what it was built to do, quickly and reliably, to the wrong answer.

Or a planner and executor wired into a loop. Planner returns in four seconds, executor in eleven, error rate zero, no timeouts. The pair has been running for two hours and has made no progress at all. Every panel is green because the failure lives between the components and no component's metrics contain it.

The pattern to internalise: a green dashboard during a failure is not a broken dashboard. It is a dashboard answering the question it was asked. The mistake was in the question.

The four instruments most systems are missing

If you want the second dial, these four are the highest return per hour of work, roughly in order of cheapness.

A count of the fallback path. How many times yesterday did a timeout, a retry exhaustion, or an exception handler return a default value instead of a real one? These paths return silently by design, so the model looks bad when it was handed a zero or never ran at all. If nobody is counting, that absence is itself the finding, and the counter usually takes an hour.

The share of runs that ended by exhausting a budget. Not the count of errors. The count of runs that hit the iteration cap or the timeout and returned something anyway. A system where a third of runs end that way is a very different object from one where none do, and most teams cannot tell you which they have.

Time since the remaining-work measure last decreased. Not time since the last activity, which is near zero in a hung system because the components are busy. Time since actual progress. This is the single metric that distinguishes hard work done slowly from nothing done energetically.

Input distribution against training distribution. The score tells you how the model does on the data you captured. The inputs tell you whether the world still looks like that. The day your real traffic stops resembling your training data is the day the score becomes a liar, and that day arrives for every system.

Why the second dial gets skipped

Worth naming, because "we should measure correctness" is obvious enough that its absence needs an explanation.

It requires domain knowledge and the first one does not. Latency is latency in every system on earth. Correctness requires you to define what right means for your specific task, maintain a labelled set, and keep it fresh. That is real work, and it is work that cannot be bought as a product.

The tooling market is lopsided. You can buy the running dial in an afternoon from any of a dozen vendors. There is no equivalent purchase for the right dial, because the hard part is your definition of correct, which no vendor has.

Green is comfortable. A dashboard that is always green is pleasant to look at and easy to report upward. An evaluation dial goes amber sometimes, which generates work and conversations. There is a quiet organisational preference for the instrument that stays quiet.

Nobody owns it. Infrastructure owns uptime. The data science team owns the model. The gap between them, whether the deployed system produces correct outputs on live traffic, frequently belongs to nobody, and the failures that live in that gap are exactly the ones that survive longest.

What to ask on Monday

Three questions, and the answers take an afternoon to get.

Which of our indicators would have moved if the system had been producing wrong answers all of last week? If the honest answer is none, you have one dial.

How often was the fallback path taken yesterday? If nobody knows, start there. It is the cheapest instrument on the list and it changes what people believe about the system more often than any other single number.

When did we last re-run the evaluation set, and on which slices? "At launch" means the number is now a historical artifact rather than a measurement.

The goal is not a second wall of panels. It is four numbers, owned by someone, reviewed on a cadence, at least one of which is allowed to go amber.

FAQ

Why didn't monitoring catch our AI failure? Because monitoring was built to answer whether the system is running, and it was running. Latency, error rate and uptime are all unaffected by whether an answer is correct, so a system can be perfectly healthy on those indicators while producing wrong output on every request.

What is the difference between monitoring and evaluation? Monitoring watches operational health: latency, errors, uptime, throughput. Evaluation watches correctness: accuracy by slice, retrieval recall, fallback rate, drift against the training distribution. They are independent, because being wrong consumes no extra resources and returns the same status code.

What should we measure that we probably are not? Four things: how often the fallback path returned a default, what share of runs ended by exhausting a budget rather than finishing, how long since the remaining-work measure last decreased, and whether today's input distribution still resembles the training distribution.

Why do most teams only build monitoring? Because it is domain-independent and purchasable, while evaluation requires you to define correctness for your own task and maintain a labelled set. There is also an ownership gap: infrastructure owns uptime, data science owns the model, and whether the deployed system is right on live traffic often belongs to nobody.

Is a green dashboard ever evidence the system is fine? It is evidence the system is running. Treat it as necessary and not sufficient. The question to test it with is whether any indicator would have moved had the outputs been wrong all week, and for most stacks the answer is no.