Four Mechanisms, Four Numbers: Why One Reliability Metric Hides the Work
Mostafa Dhouib·7 min readAn integration layer that improved API success, end-to-end failure, duplicate results and empty responses did so through four separate mechanisms. Reported as one number, you cannot tell which part earned its keep, you cannot tell when one regresses, and you cannot defend the architecture. Measure each mechanism on its own, and the gap between two of the numbers becomes the most informative thing in the report.
Four Mechanisms, Four Numbers: Why One Reliability Metric Hides the Work
The short answer. Reliability work is usually reported as one figure, because one figure is what an executive summary wants. That figure is a weighted blend of several independent mechanisms, which means it cannot tell you which one is working, it cannot alert you when one regresses while another improves, and it cannot justify the structure that produced it. On an integration layer, four distinct mechanisms produced four distinct movements: API success from 90.6 to 98.95 percent, end-to-end search failures from 8.4 to 1.2 percent, duplicate results from 12.2 to 1.2 percent, and empty results caused by provider failure from 6.8 to 0.9 percent. Reported as a single reliability score, all four collapse into one number that explains nothing. The most useful fact in that set is the gap between the first two.
There is a strong pull toward one number. It fits on a slide, it trends over time, and it answers "is it better" without requiring anyone to hold four things in their head.
The cost is that it answers that question and no others, and the others are where the engineering lives.
The four mechanisms
On the integration work above, each number moved for a different reason.
API success rate improved because retry and backoff handling started respecting each provider's actual rate-limit semantics rather than treating a limit as a generic error. A rate limit is a provider saying come back later. Treating it as a failure wastes the retry budget and often makes the limit worse. This mechanism lives entirely inside the adapter.
End-to-end failure rate improved further, proportionally, than the API number did. That gap is containment doing its work: a provider call can now fail without the user's request failing. This is a different mechanism from the first, and it is the one that matters most to a user.
Duplicate rate improved because of normalisation. Providers resell overlapping inventory under different identifiers, and two results cannot be compared until they share a shape. This is not reliability at all, it is representation, and it ended up in the same report because it was the same project.
Empty results from provider failure improved because of circuit breaking. A failing provider is cut out at its own boundary and the search returns the rest of the market rather than nothing.
Four mechanisms, four locations in the code, four different ways to regress.
What one number costs you
Attribution. If the blended score improves, which of the four earned it? You cannot tell. So the next round of work is allocated by intuition rather than by evidence, which is how teams keep investing in the mechanism that is already working.
Regression detection. Two mechanisms moving in opposite directions can leave the blend flat. A circuit breaker that stops tripping correctly while retry handling improves looks like no change at all. The composite is stable and the system is quietly worse.
Defensibility. When somebody asks whether the adapter layer was worth building, "reliability went up" is a weak answer. "Empty results caused by provider failure went from 6.8 percent to 0.9 percent because failure is now contained at the adapter" is an argument.
The gap. With four numbers, the relationships between them carry information that no individual number does, and the composite destroys exactly that.
The gap is the finding
The most interesting fact in that set is that end-to-end failures fell proportionally further than API success rose.
If a provider call failing always meant the user's request failing, those two would move together. They did not. The difference is the containment layer, and its size is a direct measurement of how much the containment is doing.
That is a number you can watch. If the gap narrows over time, containment is degrading: something is letting provider failures propagate again, perhaps a new code path that calls a provider outside the adapter. Nothing in a blended reliability score would show that.
This generalises. In any layered system, the relationship between a lower-level failure rate and the user-visible failure rate measures how much the layers between them are absorbing. Tracking both and watching the ratio is more informative than either alone.
How to report it
One number per mechanism, with its mechanism named. Not "duplicate rate" but "duplicate rate, from cross-source deduplication after normalisation". The name makes the owner and the regression mode obvious.
Before and after on the same definition. Easy to get wrong when a measurement changes alongside the system. If the definition moved, say so and report both.
The derived relationships explicitly. The gap between provider-level and user-level failure. The share of searches degraded rather than failed. These are the numbers that describe the architecture rather than the components.
A composite only if someone genuinely needs one, built from the four rather than replacing them, and never the only thing on the page.
The counter-argument, and its limit
The case for one number is real. Four numbers on a dashboard is four things to watch, three of which are not moving most weeks, and attention is finite. A single indicator that says "look here" has genuine value.
The resolution is layering rather than choosing. A composite can be the thing that draws attention. It cannot be the thing that explains, and it must not be the only thing retained, because the moment you need to know why something moved, the components are either there or they are not.
Four numbers is also not a general prescription. The right count is the number of independent mechanisms, which you find by asking how many separate ways this could regress. If the answer is one, one number is correct. If it is four, one number is a summary of four things pretending to be a measurement of one.
The honest limit
Separating the numbers tells you which mechanism moved. It does not tell you whether the mechanism is correctly built, and it says nothing about the failures none of the four cover.
A provider returning well-formed, confident, wrong data appears in none of them. API success counts it as a success, containment has nothing to contain, deduplication happily merges it, and the circuit breaker sees a healthy provider. That failure needs cross-source comparison, which is a fifth mechanism and a fifth number, and knowing it is absent is worth more than any of the four.
FAQ
Why is a single reliability score a problem? Because it blends independent mechanisms, so it cannot say which one moved, it cannot detect two mechanisms moving in opposite directions, and it cannot justify the architecture that produced it. Improvements get allocated by intuition rather than evidence, which tends to mean more investment in whatever is already working.
How many metrics should a reliability report have? As many as there are independent mechanisms. Find the number by asking how many separate ways the system could regress. One mechanism means one number is right. Four mechanisms mean one number is a summary of four things pretending to be a measurement of one.
What does the gap between provider failure and user-visible failure mean? It measures how much the layers in between are absorbing. If a provider call failing always meant a user request failing, the two rates would move together. The difference is containment, and watching the ratio tells you when containment is degrading, which no blended score reveals.
How should I name a reliability metric? After its mechanism, not its symptom. Not duplicate rate but duplicate rate from cross-source deduplication after normalisation. The mechanism name makes the owner, the regression mode and the thing to investigate obvious when the number moves.
Is a composite metric ever useful? As an attention signal, built from the components rather than replacing them. It can tell you to look. It cannot explain, and the components have to be retained, because the moment you need to know why something moved they are either there or they are not.
What do these four numbers miss? A provider returning well-formed, confident, wrong data. API success counts it as a success, there is nothing for containment to contain, deduplication merges it happily, and the circuit breaker sees a healthy provider. Catching it needs cross-source comparison, which is a separate mechanism and a separate number.
Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.
Bring us the program