Temperature Zero Is Not Deterministic

Mostafa Dhouib·10 min read

Temperature Zero Is Not Deterministic

The short answer. Event sourcing rests on one assumption. Replay the source events through the same derivation and you get the same derived state. Put a model call inside that derivation and the assumption fails, because the lowest temperature setting reduces variation without eliminating it, and the providers say so in their own documentation. That is not a bug you can configure away. It means a model-derived layer cannot be regenerated by replay, so it is not really a derived layer at all. The resolution is to stop planning to recompute it. Store the model output and the model version as part of the record, and treat that stored output as the fact.

I keep meeting the same design. An append-only event log, a projection that rebuilds derived state from the log, and somewhere in the projection, a call to a hosted model that turns unstructured input into structured fields. The team is pleased with it, and the architecture diagram is correct in every respect except one.

The assumption you are actually relying on

Event sourcing is a trade. You give up the convenience of editing state directly, and in exchange you get a record where the history is primary and the current state is a consequence. The thing that makes the trade worth it is replay.

Replay is what lets you fix a bad projection without data loss, rebuild after a corruption, run a new projection over old events, and prove that your current state follows from your recorded history. Every one of those benefits depends on the derivation being a deterministic function from event sequence to state. Same events in, same state out, today and in three years.

Nobody writes that assumption down, because for ordinary code it is nearly free. Pure functions over immutable inputs are the default, and the ways you break determinism are well known and visible in review. Someone reads the clock. Someone iterates a hash map. Someone generates a random identifier. These get caught, because practitioners know to look for them.

A model call is the same category of mistake with none of the visibility. It looks like a function call. It is not one.

Why the lowest temperature setting does not give you determinism

The common belief is that temperature is the randomness dial, so turning it to its minimum turns randomness off. Temperature does control the sampling distribution, and setting it as low as the interface allows does make the output far more stable than a higher setting. It does not make it fixed, and the people who run these services document that it does not.

The mechanism is worth understanding, because it explains why this is not a defect waiting to be patched.

Sampling is only one source of variation. Lowering temperature narrows the sampling step toward always taking the highest-probability token. It does nothing about the probabilities themselves, which are the output of a large floating-point computation.

Floating-point addition is not associative. Change the order in which partial results are summed and the final value can differ in the last bits. Inference engines batch requests together, and the batch composition depends on what other traffic arrived at the same moment. Different batching means different reduction orders means slightly different logits for your request.

Slight differences in logits change the argmax when two tokens are close. Most of the time the top token wins by a wide margin and the difference is invisible. Occasionally two continuations are nearly tied, the last-bit difference flips which one is on top, and the generation diverges from that token onward. One flipped token early in a structured output can change the whole shape of the result.

The serving fleet is heterogeneous and the serving stack changes. Different hardware generations, different kernel implementations, different library versions and different parallelism layouts all produce numerically different results for the same mathematical operation. You do not control which machine serves your request, and you are not told.

This is why provider documentation describes low temperature as more deterministic or nearly deterministic rather than deterministic, and why features intended to improve reproducibility are usually described as best effort. The honest summary is that you can get high stability and you cannot get a guarantee. Confirm the current wording in your provider's own documentation before you write anything into a design document, because the terminology and the available controls change.

What that does to the architecture

Combine the two facts and the conclusion is forced.

Replay requires a deterministic derivation. A model call is not a deterministic derivation. Therefore a layer derived through a model call cannot be reconstructed by replay.

This is the exact situation described in the companion piece on how a derived layer silently becomes a source. The model-derived layer holds information that cannot be regenerated from the event log, which makes it a source layer by the only definition that matters, whatever the diagram says.

The practical failure mode is not subtle once you look for it. A team rebuilds a projection after a schema change. Most records come back identical. A small number come back different, because the model made a different call this time on borderline inputs. In a regulated record, those are not cosmetic differences. The extracted field that drove a decision last year now has a different value, nothing in the audit trail explains the change, and you cannot tell whether the old value or the new value is the one that was used.

Worse, you have now lost the ability to answer the question an auditor will ask. What did the system conclude at the time, and on what basis? If the answer is "we would have to run it again and see", you do not have a record.

The resolution: store the output, not the recipe

The fix is to stop treating the model output as something you compute and start treating it as something that happened.

Write the model output into the source layer as a fact. The event is not "document received, extract fields on demand". The event is "extraction produced this result at this time". That event is append-only like every other source fact, and it is never recomputed.

Store the model version alongside it, as part of the same record. Not the family name, the specific version identifier the provider exposes. Without it, the stored output is an orphan, because you cannot say what produced it.

Store the full request context that determined the output. The prompt template and its version, the input as it was actually sent after any preprocessing, the parameter values including temperature, the tool or schema definitions in force, and any retrieved context that was included. The question you are preserving the ability to answer is "why did it say this", and the answer is made of these parts.

Record the output verbatim before any parsing. Store the raw response as well as the structured fields you extracted from it. When the parse turns out to be wrong, the raw response is the evidence that lets you fix the parse without re-running the model.

Make downstream layers derive from the stored output, not from the model. Everything after the model call can and should be deterministic. Validation, normalisation, scoring and aggregation all replay cleanly if they read the stored extraction rather than calling out again. Keep the nondeterministic step as thin and as early as possible, so that the non-replayable part of your record is one well-marked layer rather than a property spread through the system.

Treat re-extraction as a new fact, never as a correction. When you improve the prompt or move to a newer model and want the better output, run it, and append the new result as a separate event with its own version metadata. The old conclusion stays in the record, because the old conclusion is what the system acted on. Deciding which extraction is current is a job for the current-state layer, which is allowed to be revisable precisely because it keeps its own history.

This also puts you in the right place for the problem that arrives on the vendor's schedule rather than yours. Models get retired, and a validated system whose behaviour depends on a model that no longer exists has a change-control problem, which is the subject of why model retirement is a regulatory event. If you stored the output and the version, retirement costs you the ability to produce new extractions with that model. If you planned to recompute, retirement costs you the record.

The honest limit

Low temperature is genuinely stable, and for a lot of systems the practical rate of divergence is low enough to ignore. If you are summarising internal notes for human readers, this article is over-engineering. The argument applies when a model output enters a record that someone relies on, is audited against, or makes a decision from.

Storing everything is not free. The request context for a retrieval-augmented call can be far larger than the output, and keeping it for every call across a retention period that may run for years is a real storage and cost decision. The usual compromise is to store hashes and template versions rather than full payloads, and to keep full payloads only for calls that feed governed records. That compromise weakens the audit position, and you should make it deliberately rather than discover it later.

There are also real efforts to make inference reproducible, including batch-invariant kernel implementations and deployment modes with tighter guarantees. Where those are available and verified they change the picture, and they are worth asking a provider about. Treat any such guarantee as something to establish in writing for your specific deployment and to re-verify when your deployment changes, not as a property of model calls in general.

Finally, this article is about reproducibility, not correctness. A stored model output is auditable and still wrong. The need for independent verification of what the model concluded does not go away because you wrote it down carefully.

FAQ

Does setting temperature to zero make a model call deterministic? No. It narrows sampling toward always choosing the highest-probability token, which makes output much more stable, but the probabilities themselves come from floating-point computation whose result depends on batching and hardware. Providers describe the lowest setting as nearly deterministic rather than deterministic. Check your provider's current wording before relying on it.

Why is this a problem for event sourcing specifically? Event sourcing earns its cost through replay, and replay assumes the derivation from events to state is a deterministic function. A model call inside the derivation breaks that assumption, so the derived layer cannot be reconstructed from the log. It holds information that exists nowhere else, which makes it a source layer rather than a derived one.

What exactly should I store for each model call? The raw output verbatim, the structured fields you parsed from it, the specific model version identifier, the prompt template and its version, the input as sent, the parameter values, any schema or tool definitions in force, and any retrieved context included in the request. That set is what lets you answer why the system reached a conclusion without re-running anything.

Why does floating-point arithmetic cause output to vary? Addition in floating point is not associative, so summing the same values in a different order can change the last bits of the result. Inference servers batch requests, and the batch depends on concurrent traffic, so reduction order varies between runs. When two candidate tokens are nearly tied, a last-bit difference flips which one is selected and the generation diverges.

Should I re-run extraction when a better model becomes available? You can, but append the result as a new fact with its own version metadata rather than overwriting the old one. The earlier conclusion is what the system acted on, so it belongs in the record permanently. Which extraction counts as current is a decision for the revisable current-state layer, which keeps its own history of that choice.

Does a reproducible-inference feature solve this? Sometimes, for a specific deployment, and only as far as the provider will commit in writing. Batch-invariant implementations and tighter deployment modes exist and are worth asking about. Features described as best effort are not a basis for a replay guarantee, and any commitment you do get has to be re-verified when your deployment type or provider terms change.