A platform handling more than 60,000 requests a day. The accuracy was fine. The monthly bill was not. Near-identical requests were computed from scratch every time, through repeated prompts, embeddings, retrieval and reasoning, with no caching anywhere along the path. At that volume the inference bill, and not the accuracy, was what threatened the product.
Every request to a model pays for the whole computation again, unless something upstream remembers the answer.
A language model keeps no memory between requests. Each call is computed and billed in full, even when it is almost identical to one answered a moment earlier. An AI feature is also rarely a single call. A typical request passes through several expensive stages: turning text into an embedding, searching a vector index for related material, assembling a prompt, running the model, and sometimes reasoning further over the result. Each stage costs time and money, and each will happily repeat work it has only just done.
Caching is the general answer to repeated work: store a result and serve it again when the same question comes back. The subtlety with AI workloads is that the same question has several meanings. An exact cache catches byte-for-byte repeats. A semantic cache catches requests that are worded differently but mean the same thing, by comparing their meaning, and it trades a little exactness for a lot of reach. Between those sit caches for the intermediate stages: an embedding already computed for this text, a prompt whose opening section has already been processed, a vector search already run. Each catches a different shape of repetition, so layering them catches more than any one could alone.
Not all the cost of an AI system is in the model. The code that moves requests through queues, retries them and fans them out can become a bottleneck of its own at high volume, especially when it is written in a scripting language chosen for convenience. Those languages are good at orchestrating calls and poor at doing heavy work on every message at high throughput. Moving the throughput-bound parts into a compiled language, and leaving orchestration where it is, puts each job in the tool that suits it.
The trouble looked like a question of accuracy or model choice, because those are the levers everyone knows. It was a question of cost architecture: nothing caught near-identical work before it reached a model. Tuning prompts on a system with no cache lowers the price of each redundant computation and leaves the redundancy exactly where it was.
AI costs run away through repetition, and the usual fixes make each repeat cheaper while leaving the repeats in place.
A feature that works well in a pilot is billed per call. When usage grows, the bill grows with it in a straight line, because nothing in the design distinguishes a new question from one answered moments ago. Teams notice the bill and reach for the levers they know: shorter prompts, a smaller model, a cheaper provider. Each lowers the unit price. None reduces the number of units, and in many workloads a large share of those units are near-duplicates.
Caching brings failures of its own when it is added carelessly. A semantic cache with too loose a sense of sameness serves a confident answer to a question that differed in a way that mattered. A cache nobody measures can stop working after an unrelated change and go unnoticed, quietly sending everything to the model again. And a queue written for convenience can cap throughput long before the model does, so the system is slow for reasons that have nothing to do with AI.
The reflex, and the fix.
Optimise the prompts
It is the accessible lever, it needs no architectural change, and it shows an immediate improvement on every call.
It lowers the cost of work that should never have been repeated. The savings are capped by the price of a call, and the problem was the number of calls.
Optimise the architecture around the models
A layered cache across the request path, and a rewrite of the high-throughput queue in a compiled language.
Requests turning back at a cache layer before they ever reach a model.
Attack the redundancy and leave the unit price for later, because the redundancy is where the bill lives.
A layered cache across the request path
In-memory, then semantic, then embedding, then prompt prefix, then vector lookup, then response. Each layer catches a different shape of near-identical work, and only what escapes all six reaches a model.
The queue rewritten for throughput
High-throughput queue processing moved to a compiled language. The scripting layer was kept only for inference orchestration, which is the job it is actually good at.
Tracing before and after
A cache that is not measured can be silently missing. A hit rate is the only evidence that the architecture is doing what it claims, so the request path was traced before the work began and again after it.
What each choice cost, and what it bought.
AMeasuring the redundancy first
It is possible to add caches on instinct. Tracing the request path before building anything costs time up front and saves nothing by itself. It shows how much of the work is actually repeated and at which stage, and that sets the ceiling on what caching can achieve. The gains here are proportional to the near-identical work that exists, so the measurement is what made the architecture worth building.
BLayering caches along the request path
A single response cache is simpler and catches exact repeats. It misses everything else: requests worded differently, requests that share an embedding or a retrieval step, requests with a common prompt prefix. Six layers mean more moving parts and more to monitor. In return each kind of near-identical work is caught at the stage that recognises it, and only what escapes every layer reaches a model.
CChoosing semantic reuse per class of request
A semantic cache gives the largest savings and carries the largest risk, because it answers a new request with a result computed for a similar one. Turning it on everywhere would have maximised the hit rate. Choosing it deliberately for each class of request means some classes give up savings in order to stay exact. That is the trade that keeps a cost reduction from turning into a correctness problem.
DMoving the queue to a compiled language
Rewriting working code is expensive and carries risk of its own. The high-throughput queue was doing heavy per-message work in a scripting layer that suits orchestration and struggles with volume. Moving that work to a compiled language, and keeping the scripting layer for inference orchestration, put each part in the language that fits it. Together with the cache, it is what roughly doubled throughput.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
Figures confirmed as proven. The rendered client quote associated with this engagement is deliberately not published.
Shorter prompts, a slightly cheaper call, and the same volume of redundant computation, with the bill still growing in step with usage.
A layered cache across the request path plus a compiled-language rewrite of the high-throughput queue, so near-identical work is served from cache and the scripting layer is kept for inference orchestration. 48 percent fewer requests to the model, 44 percent lower average latency, 58 percent lower monthly inference cost, and roughly twice the throughput, with the savings paying for the project within weeks.
What it owns, and what it hands to a person.
A semantic cache trades a small amount of exactness for a large amount of cost, and that trade has to be chosen deliberately for each class of request. Applied globally, it would be the wrong trade somewhere. The gains are proportional to how much near-identical work exists, which is why the measurement came before the architecture.
When a bill grows with usage and the accuracy is already fine, the cause is almost never the model. Measure how much of the work is repeated before optimising the price of doing it.
When accuracy is fine and the bill grows with usage, look for repeated work before looking at the model or the prompts.
Cheaper calls lower the unit price. Caching lowers the number of calls, and at high volume the number is usually where the money is.
Different caches catch different kinds of repetition, from exact repeats to shared intermediate steps, so layering them reaches further than any single cache.
Semantic caching trades exactness for cost. Decide per class of request where that trade is acceptable, and keep the rest exact.
A cache you do not measure can stop working without anyone noticing. Trace hit rates before and after every change.