Express the cost in the unit your customer buys: a seat, a document, a ticket, an account. Per request it stays comfortably small and tells nobody anything, which is why the number is usually only discovered when the bill arrives.
| Component | Cost | Multiplier | Share |
|---|---|---|---|
| Base model call | $0 | 1.0x | 66% |
| Nested calls | $0 | 1.40x | 26% |
| Retries | $0 | 1.08x | 7% |
| Cost per request | $0 | 1.51x | 100% |
Context grows without anyone deciding it should. Prompts accumulate instructions, retrieved context grows with the corpus, and a feature costed at launch frequently runs at several times that a year later with no decision ever having been made.
Cost per thousand requests, by context size
One call per request, 600 output tokens, no retries and no nesting, so these are floors rather than estimates. The shape is the point: input tokens dominate once retrieval is in the path, which is the opposite of where most teams look when the bill surprises them.
| Input tokens | Economy (0.3/1.2 per M) | Mid (3/15 per M) | Frontier (15/75 per M) |
|---|---|---|---|
| 1,000 | $1.02 | $12 | $60 |
| 2,000 | $1.32 | $15 | $75 |
| 4,000 | $1.92 | $21 | $105 |
| 8,000 | $3.12 | $33 | $165 |
| 16,000 | $5.52 | $57 | $285 |
| 32,000 | $10 | $105 | $525 |
Prices are illustrative tiers rather than any vendor's current rate card; substitute your own. Amber is over $20 per thousand requests, red over $100. A feature at 40 requests per seat per month on the mid tier at 8k context costs about $1.32 per seat before retries.
A worked example
A SaaS product with an AI assistant, priced at 29 dollars per seat per month. Each request sends about 3,200 input tokens and returns 600, on a mid tier. Usage runs at roughly 40 requests per seat per month. At launch somebody worked out that a request costs about a cent, which is plainly fine.
It is fine, but the number they checked was the wrong one. Eight percent of requests retry, and the assistant can invoke a second model call for retrieval planning about forty percent of the time, so the real multiplier is 1.51x on every request. Cost per seat is $1.12 against 29 dollars, which is a 96 percent gross margin on the feature. Still fine, and now it is fine for a reason somebody could defend.
The part worth acting on is the drift. Prompts accumulate instructions and retrieved context grows with the corpus, so 60 percent context growth over a year is unremarkable and nobody will ever approve it. That takes cost per seat to $1.47 and the margin to 95 percent. On these numbers that is comfortable.
On different numbers it is not, and the reason to run this early is to find out which you are. A feature at 400 requests per seat instead of 40 has the same shape and a tenth of the headroom, and the same 60 percent drift takes it somewhere that needs a decision rather than a note.
The arithmetic, so you can check it
Cost per call is (input tokens / 1M) x input price + (output tokens / 1M) x output price. Cost per request multiplies that by the nested-call factor and by one plus the retry rate. Cost per unit sold multiplies again by requests per unit, and the margin is (price - cost) / price.
This prices inference and nothing else. Vector database queries, reranking, embedding at ingest, storage and the engineering time to keep it working are all real and none of them are here. It is also a mean rather than a distribution: a feature whose p99 request costs twenty times its median has a tail risk this cannot see. Use it to establish the shape and whether the margin survives drift, then add the rest.
What happens to the rest of the system at that volume is in what happens to your AI feature at ten times the traffic. For the tree bound specifically, use the agent reliability calculator.
Questions
Because every inference costs what it costs. There is no equivalent of a warmed cache for a novel request, so unlike an ordinary web feature where fixed costs spread and caches warm, ten times the volume is ten times the bill plus whatever retries add. Unit economics that were a rounding error at pilot volume become the largest line item on the day the feature succeeds.
Per unit your customer buys: per seat, per document, per ticket, per account. That is the only form in which it can be compared against your price. Expressed per request it stays comfortably small and tells nobody anything, which is why the number is often only discovered when the bill arrives.
Three things, reliably. Retries, because every retry is a full-price inference and the requests that need them are the ones that need them most. Context growth, because prompts accumulate and retrieved context grows with the corpus, with no decision ever being made. And nested calls, because if any part of the feature can invoke another model call then one user action is not one inference.
With a per-request budget covering total inference spend and depth, carried through every nested call and enforced outside any individual call. Bounding a single call bounds nothing about the tree: twenty capped calls each invoking twenty capped calls is four hundred inferences, every one inside its own limit.
That depends on what else the unit has to carry, but the shape matters more than the threshold. A margin that only works at today's context size and retry rate is not a margin, it is a snapshot, because both of those drift upward on their own. The defence is a measured token budget per request with an alert on it, rather than an intention to keep prompts short.