Output Tokens Are The Bill: Why Trimming Your Prompt Did Not Help

Mostafa Dhouib·10 min read

Output Tokens Are The Bill: Why Trimming Your Prompt Did Not Help

The short answer. For most model-backed features, the cost sits on the output side. Output tokens are priced at a multiple of input tokens, and once you cache a stable prompt prefix the repeated input gets cheaper again, which widens the gap further. Yet the standard cost reduction exercise is to shorten the system prompt and trim retrieved context, because input is the part a team can see in its own code and control directly. Output is produced by the model and feels like something that happens to you. It is not. Output length is a design decision, and it is also what sets latency, which compounds when a slow answer turns into a timeout and a retry.

The conversation usually starts the same way. The bill went up, someone profiled the prompt, found a system prompt that had grown over months, and cut it in half. The bill moved a little. Everyone concluded that models are just expensive.

Nothing in that sequence is unreasonable. It is just aimed at the smaller half of the problem.

Where the cost actually sits

Three things stack up to put the weight on the output side.

Output tokens are priced higher than input tokens. Across the current model lineups this is not a marginal difference. It is a multiple, and it holds at every tier from the cheapest small model to the largest. A thousand tokens of answer costs several times what a thousand tokens of question costs.

Caching makes repeated input cheaper again. A stable prompt prefix, the system instructions and the tool definitions, can be cached and read back at a fraction of the price of fresh input. If your feature sends the same large preamble on every call, the second call onwards is paying very little for it. There is no equivalent mechanism on the output side, because every answer is new.

Reasoning tokens are output tokens. Where a model thinks before answering, that thinking is billed as output whether or not you display it. A feature that produces a two-line answer after extensive internal reasoning has a large output bill and a small visible response, which is exactly the case people misread as cheap.

Put those together and the common profile is a feature whose input is large but heavily cached and cheaply priced, and whose output is small in word count but expensive per token and often invisible. The visible part of the system is the part that does not matter much.

Why everyone optimises the wrong side

This is a visibility problem rather than a competence problem.

The input is in your repository. You can read the system prompt, count its tokens, see the retrieved chunks, and decide to send fewer of them. It is a file you can edit. There is a satisfying, measurable action available.

The output is not in your repository. It arrives. The only levers on it are indirect: instructions about brevity, a format constraint, a cap on maximum tokens, a different effort setting. None of those feel like engineering in the way deleting three paragraphs from a prompt does, and some of them feel risky, because constraining the answer might make it worse.

There is also a measurement artefact. Most teams log input tokens carefully, because they are computed before the call and used for context-window management. Output tokens come back in the response and often get logged as a single total rather than attributed to a feature. So the input side has a dashboard and the output side has a monthly invoice.

Long outputs are slower in a way that compounds

The cost argument is the easy half. The latency argument is the one that changes architectures.

Input is processed in parallel. Output is not. Tokens are generated one at a time, each conditioned on the ones before it, so the time to produce an answer scales with its length in a way that the time to read a prompt does not. Doubling the context does not double the wait. Doubling the answer roughly does.

That alone is a user experience problem. The compounding is what makes it a reliability problem.

Long answers hit request timeouts. A non-streaming request with a generous token cap can exceed the client's timeout. The work was done and billed. The response was discarded.

A timeout becomes a retry. The retry re-sends the same request and generates the same long answer, so you pay twice for one result, and the second attempt is no more likely to finish inside the timeout than the first.

A truncated answer becomes a retry too. Hit the maximum output token cap and generation stops mid-answer. If the output was structured, it will not parse, so the pipeline retries. You have now paid for a full long generation, got nothing usable, and paid for another one.

In an agent loop, every step's output becomes the next step's input. A verbose intermediate step inflates the context of every subsequent step in the same run. The expensive output tokens are charged once, and then the same text is charged again as input on every following call. Verbosity early in a loop is the single most expensive habit we see.

That last one is why cost per call is such a misleading metric. The call that caused the expense is not the call that shows it.

Structured output as a cost control

Everyone treats structured output as a correctness feature, a way to get parseable data back. It is also the most effective output cost control available, for a reason that has nothing to do with parsing.

A schema removes the parts of an answer that nobody asked for. No preamble restating the question. No closing summary. No explanation of the format. No hedging paragraph. For an extraction or classification feature, those unrequested sections are frequently most of the output.

Three practical notes from doing this.

Constrain the fields, not just the shape. A schema with a free-text field in it has a schema-shaped wrapper around an unconstrained answer. If a field should be one of five values, make it an enumeration. If it should be a short phrase, say so and bound it.

Separate the decision from the explanation. Many features ask for a verdict plus the reasoning behind it, then show only the verdict. If the explanation is only read when someone disputes the verdict, generate it on demand for the disputed cases rather than on every call.

Watch the interaction with reasoning. Constraining the visible output does not constrain internal reasoning, which is billed on the same side. A terse schema with a high effort setting can cost more than a prose answer at a low one. Both levers are output levers and they need tuning together.

Measure cost per operation, not per call

The metric that makes all of this tractable is cost per completed operation: one resolved support ticket, one document extracted, one pull request produced. Not cost per request.

The difference is everything the per-call number hides. Retries after a timeout. Retries after a truncated structured response. Multiple turns in an agent loop. Follow-up calls because the first answer was too vague to act on. A cheaper per-call configuration that needs three attempts is not cheaper.

Two things to put in place.

Attribute tokens to a unit of work, not a call. Tag every request with an operation identifier that survives retries and loop iterations, then sum input, cached input, and output tokens per operation. The resulting distribution is more informative than the mean, because cost problems in model-backed systems are almost always a long tail of operations that went around the loop too many times.

Log the reason generation stopped. Every response says whether the model finished, hit the token cap, or stopped for another reason. Counting the cap-hit rate per feature tells you immediately where you are paying for truncated work, and it is a one-line addition to a response handler.

The honest limit

The shape described here, output dominating cost, holds for most of the model-backed features we work on, and it does not hold for all of them. A feature that stuffs a very large uncached document into every call and asks for a one-word answer is genuinely input-dominated, and for those the prompt-trimming instinct is correct. Long-context retrieval pipelines with poor cache hit rates are the main real exception. The way to know which you are is to measure, not to assume either way.

We have deliberately not given pricing figures or ratios here, because they differ by provider and tier and change without notice. The structural facts are that output is priced above input, that cached input reads cost a fraction of fresh input, and that generation is serial while input processing is parallel. Those are what the argument rests on, and they are stable. The multiples are not.

The uncomfortable part is that constraining output is a quality lever as well as a cost lever, and we cannot tell you where your threshold is. A tighter schema and a lower effort setting will reduce spend. Somewhere below that, the answers get worse. Finding that point requires an evaluation set for your own task, and a team without one is choosing between guessing and overspending.

FAQ

Why did shortening my prompt not reduce my model costs? Because the cost was mostly on the output side. Output tokens are priced at a multiple of input tokens, and a stable prompt prefix can be cached and read back at a fraction of fresh input cost, so repeated input is cheap. Trimming a system prompt that was already being served from cache changes very little. The lever that matters is the length and shape of the answer.

Are output tokens really more expensive than input tokens? Yes, by a multiple rather than a small margin, and this holds across model tiers. Caching widens the gap further, because there is a mechanism to make repeated input cheap and no equivalent for output, since every answer is newly generated. Reasoning tokens count as output too, even when they are never displayed.

Why are long responses slower, not just more expensive? Input is processed in parallel but output is generated one token at a time, each conditioned on the previous ones. Doubling the context does not double the wait. Doubling the answer roughly does. That makes long answers a timeout risk, and a timeout becomes a retry that regenerates the same long answer at full cost for no result.

How does structured output reduce cost? A schema removes the output nobody asked for: the restated question, the closing summary, the explanation of the format, the hedging. On extraction and classification work those sections are often most of the response. Constrain the individual fields as well as the overall shape, using enumerations where the answer is one of a known set, or you have a schema wrapped around free text.

Why measure cost per operation instead of per call? Because the per-call number hides retries after timeouts, retries after truncated structured responses, multiple turns in an agent loop, and follow-up calls caused by a vague first answer. A configuration with a lower per-call cost that needs three attempts costs more per finished piece of work. Tag requests with an operation identifier that survives retries and sum tokens per operation.

What is the most expensive habit in an agent loop? Verbose intermediate steps. Each step's output becomes part of the next step's input, so a long intermediate answer is billed once at the higher output rate and then again as input on every subsequent call in that run. Keeping intermediate outputs terse and structured is worth more than any change to the initial prompt.