Durable Execution Is Not Exactly-Once (and What to Do Instead)

Mostafa Dhouib·September 4, 2026·7 min read

Durable Execution Is Not Exactly-Once (and What to Do Instead)

The short answer. Durable execution persists a workflow's history and, on resume, replays it while skipping the steps that already completed. That is genuinely valuable and it narrows the window in which a side effect can run twice. It does not close it, because performing an effect and recording that you performed it are two operations that cannot be made atomic across a network boundary. Nothing provides exactly-once. What you actually get is at-least-once delivery plus a mechanism for making the second execution harmless, and that mechanism is yours to build.

Durable execution frameworks are good, and I recommend them. This article exists because of what teams believe after adopting one.

The belief is that the framework now guarantees each step runs exactly once, and therefore idempotency is a problem that has been handled. That belief is wrong, and it is dangerous specifically because it removes the motivation to build the thing that actually protects you.

What durable execution actually does

The mechanism is worth stating plainly, because the guarantee follows directly from it.

The framework records an event history as your workflow runs. Each completed step and its result are appended to that history and persisted. If the worker dies, or the process is cycled, or the workflow is resumed for any reason, the framework replays your code from the beginning and, for each step it finds already recorded, returns the stored result instead of executing it again.

That is real and it is useful. It means a crash halfway through a twenty-step workflow does not redo the first nineteen steps. It gives you resumability across process death, which is hard to build and easy to get wrong.

Note what the guarantee is about: it is about replay. Steps that were recorded as complete are not re-executed on replay. That is a statement about the framework's own bookkeeping.

Where the window is

Now look at what a step actually does when it performs a side effect.

It calls the external system. The external system does the work. It returns a response. The framework records the result in the history.

Between "the external system did the work" and "the framework recorded it" there is a gap. It is small. It is not zero, and it cannot be made zero, because those are two different systems and there is no transaction spanning both.

If the worker dies inside that gap, the effect happened and the history does not know. On replay, the framework finds no record of the step, and runs it again.

That is the whole argument, and it is not a criticism of any implementation. It is a consequence of the fact that you cannot atomically commit a change in someone else's database and a change in yours. It is the same reason a timeout is ambiguous: the information you would need is not in your possession.

There are second-order versions too. A step that returns a value the framework cannot serialise deterministically. Non-deterministic code that takes a different path on replay. A worker that is partitioned rather than dead, and continues running while a replacement replays the same workflow, which is the redispatch door in a different costume.

The honest characterisation

What durable execution gives you is not exactly-once. It is:

At-least-once execution of each step, with the probability of a second execution reduced substantially.

That reduction is worth paying for. A window measured in milliseconds around a crash is much better than an unbounded one. But "much less likely" and "cannot happen" are different engineering premises, and at production volume the difference is the number of incidents per year rather than zero.

The rule of thumb I use: if the consequence of a second execution is unacceptable, the probability being small does not help you. Rare multiplied by catastrophic is still a project you will be doing at an inconvenient time.

What to do instead, and it is not much work

The framework is not the protection. It is the substrate. The protection is the same one that has always worked.

Make the step idempotent anyway. Generate a stable idempotency key when the action is first conceived, derived from the workflow identifier and the step identity rather than randomly per attempt, and pass it to the external system. This is the whole fix, and it is worth being explicit that the key must be a function of the intent, because a fresh key per attempt makes every replay look like a new action and defeats the mechanism entirely.

Prefer absolute writes over relative ones. "Set the state to this" is safe to repeat. "Change it by this much" double-counts. Wherever you can express an effect as a desired state rather than a delta, repetition stops mattering, which is how fleet systems made this class of bug structurally impossible.

Put the irreversible step last. After everything that could still fail has already succeeded. A duplicate of the last step is one problem; a duplicate mid-workflow with fifteen steps of consequences after it is another.

Do not rely on compensation as the primary defence. Compensating transactions are useful and they are strictly weaker than prevention. They require the compensation itself to succeed, they leave the effect visible in between, and for some effects there is no compensation at all. You can refund a charge. You cannot un-send an email, and the customer already read it.

Keep the reconciliation check. Count what the workflow intended against what actually executed against the external system. This is the same intended-versus-executed check that catches dropped tool calls, and it catches the other direction too.

What to ask before adopting one

Three questions that get a straight answer from a good vendor and evasion from a bad one.

What is the exact guarantee, stated in terms of at-least-once or at-most-once, rather than in terms of reliability? Any answer containing "exactly-once" without a qualifier about idempotent effects is marketing.

What happens if a worker dies after an external call succeeds and before the result is recorded? A good answer describes the window and tells you to make the step idempotent. That is the correct answer, and hearing it is a reason to trust the rest of what they say.

How do you handle a partitioned worker that is not dead? This is the redispatch case, and it is the one that produces two concurrent executions rather than two sequential ones, which breaks weaker mitigations.

The general shape

This is the same lesson as almost everything else in distributed systems, wearing a new label. The guarantee is at-least-once, the second delivery is normal rather than exceptional, and the thing that saves you is making the second one harmless rather than preventing it.

Every side effect your system takes is either idempotent or it is a bug waiting for a retry. Durable execution changes how often the retry arrives. It does not change which of those two categories your effect is in.

FAQ

Does durable execution guarantee exactly-once? No. It records completed steps and skips them on replay, which substantially reduces the chance of a second execution. It cannot eliminate it, because performing an external effect and recording that you performed it are two operations with no transaction spanning both. The honest characterisation is at-least-once with a much smaller window.

Where exactly is the window in a durable workflow? Between the external system completing the work and the framework persisting that result to its history. A worker that dies in that gap leaves an effect that happened and a history that does not know, so replay runs the step again.

If I use a durable execution framework, do I still need idempotency keys? Yes, and this is the main practical takeaway. Generate a stable key derived from the workflow and step identity rather than randomly per attempt, and pass it to the external system. A fresh key per attempt makes every replay look like a new action.

Are compensating transactions a substitute for idempotency? No, they are strictly weaker. They require the compensation to succeed, they leave the effect visible in the interval, and some effects have no compensation. Use them as a backstop, not as the primary defence, and put irreversible steps last.

What should I ask a durable execution vendor? State the guarantee in at-least-once or at-most-once terms rather than reliability terms, describe what happens when a worker dies after an external call succeeds but before the result is recorded, and explain the handling of a partitioned worker that is still running. Good answers to all three tell you to make steps idempotent anyway.