A platform handling more than 60,000 requests a day where the accuracy was fine but the monthly bill was not, because near-identical requests, repeated prompts, embeddings, retrieval, and reasoning, had no caching anywhere. Runaway inference cost on high request volume, where the bill rather than the accuracy is what threatens the product.
The stated problem reads as accuracy or model choice, because those are the levers everyone knows. The real problem is cost architecture: the absence of caching around near-identical work. Optimizing prompts on a system with no cache reduces the price of each redundant computation without reducing the redundancy.
The reflex, and the fix.
Optimize the prompts
It is the accessible lever, it needs no architecture change, and it shows an immediate per-call improvement.
It lowers the cost of work that should never have been repeated. The savings are bounded by the price of a call, and the problem is the number of calls.
Optimize the architecture around the models
A layered cache across the request path and a compiled-language rewrite of the high-throughput queue.
Most requests turning back at a cache layer before they ever reach a model.
Attack the redundancy rather than the unit price, because the redundancy is where the bill lives.
A layered cache across the request path
In-memory, then semantic, then embedding, then prompt-prefix, then vector-lookup, then response. Each layer catches a different shape of near-identical work, and only what escapes all six reaches a model.
The queue rewritten for throughput
High-throughput queue processing moved to a compiled language, with the scripting layer kept only for inference orchestration, which is what it is actually good at.
Tracing before and after
Because a cache that is not measured is a cache that is silently missing, and a hit rate is the only evidence that the architecture is doing what it claims.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
Figures confirmed as proven. The rendered client quote associated with this engagement is deliberately not published.
Shorter prompts, a marginally cheaper call, and the same volume of redundant computation, with the bill still growing linearly against usage.
A layered cache across the request path plus a compiled-language rewrite of the high-throughput queue, so near-identical work is served from cache and the scripting layer is reserved for inference orchestration. 48 percent fewer requests to the model, 44 percent lower average latency, 58 percent lower monthly inference cost, and roughly twice the throughput, with the savings paying for the project within weeks.
What it owns, and what it hands to a person.
A semantic cache trades a small amount of exactness for a large amount of cost, and that trade has to be chosen deliberately per request class rather than applied globally. The gains are proportional to how much near-identical work exists, which is why the measurement came before the architecture.
When a bill grows with usage and the accuracy is already fine, the problem is almost never the model. Measure how much of the work is repeated before optimizing the price of doing it.