What Happens to Your AI Feature at Ten Times the Traffic

Mostafa Dhouib·September 7, 2026·7 min read

What Happens to Your AI Feature at Ten Times the Traffic

The short answer. Three things break, and they break differently from the rest of your stack. Cost is per-request with nothing to amortise, so unit economics that were rounding errors at pilot volume become the largest line item. Latency degrades through queueing at the inference tier, which produces a knee rather than a slope. And under saturation the feature returns degraded output rather than an error, so your availability metrics stay green while quality collapses. All three arrive on the day the feature becomes popular.

Ordinary web features get cheaper per request as they scale. Caches warm, connections pool, fixed costs spread.

An AI feature does the opposite in most of the ways that matter, and teams meet this at the least convenient moment: the launch that worked.

One: the cost does not amortise

Every inference costs what it costs. There is no equivalent of a warmed cache for a novel request, and at ten times the volume the bill is ten times, plus whatever the retries add.

This is arithmetic anyone can do in advance and few do, because at pilot volume the number is small enough to ignore. The move worth making early is to express the cost per unit of the thing your customer buys: per seat, per document, per ticket, per account. A number expressed that way can be compared against your price, and it either survives that comparison or it does not.

Three things reliably make the number worse than the estimate:

Retries. Every retry is a full-price inference. A three-retry policy on a flaky path is a three-times cost multiplier on the requests that need it most.

Context growth. Prompts accumulate. Retrieved context grows as the corpus grows. A feature costed at launch with a 2,000-token prompt frequently runs at several times that a year later, with no decision ever having been made.

Call trees. If any part of the feature can invoke another model call, the cost per user action is not one inference. Bounding a single call bounds nothing about the tree, which is the same structural gap that makes agent spend unpredictable.

Two: latency has a knee, not a slope

Web tiers degrade gracefully because you can add stateless replicas. Inference tiers degrade through queueing at a constrained resource, and queueing has a characteristic shape: flat, flat, flat, then a wall.

Below capacity the queue is empty and latency is service time. Approaching capacity the queue grows without bound, and each additional request adds delay to everything behind it. There is no gentle middle.

Two consequences that trip teams up.

Your load test probably missed it. A ramp that stops at twice current traffic tests the flat region and reports that everything is fine. The information is entirely in the region past the knee, and finding the knee is the point of the test.

The p99 moves long before the median. By the time the average is visibly bad, a meaningful share of users have had a very poor experience. Watch the tail, and watch queue depth at the inference tier directly rather than inferring it from response time.

The underlying constraint is often memory bandwidth rather than arithmetic throughput. If a model's weights must cross the bus for every forward pass, the ceiling is set by bytes moved per second, and adding compute does not raise it. Knowing which side of that you are on determines whether batching helps, whether a smaller model helps, and whether more hardware helps at all.

Three: saturation degrades quality, not availability

The most dangerous of the three, because your existing monitoring is structurally unable to see it.

Under pressure, the standard mitigations are: shorter timeouts, fall back to a smaller model, truncate the context, reduce the number of retrieved passages, skip the reranking step. Every one of those returns a response. HTTP 200. Normal latency, once the timeout does its job.

So the system sheds quality rather than load, and quality is the one dimension nothing is watching. Availability holds, error rate holds, latency recovers, and the answers get worse.

This is the two-dials problem arriving through a side door: monitoring answers whether the system is running, and under saturation the system is running fine while being wrong more often.

The instruments that make it visible, none of which are expensive:

Count every degradation event by type. Fallback to the smaller model, context truncated, retrieval reduced, reranking skipped. Each with its own counter. If nobody is counting, you cannot distinguish a healthy hour from a degraded one after the fact.

Report quality on a maintained evaluation set, continuously. Not at launch. Continuously, so a change in the number has a time attached.

Alert on degradation rate, not just latency. A feature serving thirty percent of requests from the fallback path is in an incident, and today nothing tells you.

What to do before the traffic arrives

Load test past the knee. The purpose is to find where it is, not to confirm the current level is fine. Ramp until it breaks, record the number, and set your capacity alarm below it.

Express the cost per unit your customer buys, including retries and current context sizes, and compare it to what you charge.

Decide the degradation ladder explicitly, in order, written down. Which quality do you shed first, second, third, and at what threshold. If this is not decided in advance it gets decided during an incident by whoever is on call.

Make degradation visible to the caller. An internal flag on the response indicating it came from a degraded path lets downstream systems and support tooling behave sensibly, and it makes the incident reconstructable.

Bound the call tree, with a per-request budget covering total inference spend and depth, enforced outside any single call.

Know whether you are memory-bound or compute-bound, because it determines which mitigations do anything.

The pattern

The unifying property is that an AI feature fails downward in quality rather than outward in errors, and every conventional scaling instrument watches for errors.

That is why a SaaS team can have a fully green dashboard during the week their feature got materially worse, and why the first credible report usually arrives from a customer rather than from monitoring.

FAQ

Why doesn't an AI feature get cheaper as it scales? Because inference cost is per-request with nothing meaningful to amortise, unlike a web tier where caches warm and fixed costs spread. Retries multiply it, context sizes grow over time without a decision being made, and nested model calls mean one user action may be many inferences.

Why does inference latency have a knee? Because the inference tier degrades through queueing at a constrained resource rather than by adding stateless replicas. Below capacity latency is service time; approaching capacity the queue grows without bound. There is no gentle middle, and a load test that stops short of the knee reports that everything is fine.

Why doesn't monitoring catch AI feature saturation? Because the standard mitigations all return successful responses: falling back to a smaller model, truncating context, retrieving fewer passages, skipping reranking. Availability, error rate and latency all stay healthy while answer quality drops, and quality is the dimension nothing is watching.

What should be measured during an AI feature load test? Where the knee is, the p99 rather than the median, queue depth at the inference tier directly, and the rate of each degradation event by type. Finding the breaking point is the purpose of the test, not confirming the current level is comfortable.

What is a degradation ladder? The explicit, ordered list of which quality you shed first, second and third under pressure, with the thresholds that trigger each step. Deciding it in advance means it is not decided during an incident by whoever happens to be on call.