Ground Truth

Why Your AI Agent Double-Executes: Retry Is a Second Execution, Not a Second Chance

Mostafa DhouibMostafa Dhouib··12 min read
The short answer

When an agent charges a card twice or sends an email twice, nothing is broken to go find: no error, no red in the logs, the action was correct and simply ran twice. Duplicates enter through four doors, retry, redelivery, redispatch, and double trigger, but a duplicate only hurts you when there is no gate in front of the side effect. The fix is one idempotency key, identical no matter which door it came through.

Why Your AI Agent Double-Executes: Retry Is a Second Execution, Not a Second Chance

The short answer. When an agent charges a card twice or sends an email twice, nothing is broken to go find, no error, no red in the logs, the action was correct and just ran twice, so the model and the prompt are not the cause. Duplicates enter through four doors, retry, redelivery, redispatch, and double trigger, but a duplicate only hurts you when there is no gate in front of the side effect. The fix is one idempotency key, identical no matter which door it came through.

An agent charges a customer three times. That customer is now in your inbox, and they are not being polite about it.

You go looking for the bug. There is no error. Nothing crashed. The logs are green from end to end. There are simply two charges sitting there, both correct, both real.

This is one of those failures that looks impossible until you understand it, and then looks inevitable. It is not a smarter-model problem and it is not a prompt problem. Let me save you the diagnosis time.

Retry is a second execution, not a second chance

Everything hinges on what a retry actually is.

You buy something online. You hit pay. It spins. Nothing comes back. Did it go through? You have no idea. So you click again, which is what any reasonable person does.

If that first click went through, you just paid twice.

The only thing standing between you and a double charge is whether the site is smart enough to look at your second click and say: I have seen this one, it is the same payment, do not charge it again. That recognition is the entire fix, and most teams never build it.

Why the machinery genuinely cannot tell

Here is what actually happened on the wire.

Your request went out. The server did the work. The money moved. The charge is real. And then the little reply saying "yes, it worked" got lost on the way home. A network hiccup, a timeout, a load balancer that gave up.

Now all you know is that you did not hear back.

It is like texting someone when your phone never shows "delivered." The message may have arrived and the receipt may have been lost. You cannot distinguish those cases from your side, and no amount of cleverness in the client will let you, because the information you need is not in your possession.

So a timeout does not mean it did not happen. It means you have no idea whether it happened.

Which makes a blind retry a coin flip. Half the time you save the day. Half the time you double-charge someone.

What you know
You did not hear back
That is the entire contents of your knowledge
What actually happened, one of two things
The request never arrived
retrying is correct and saves the day
It committed and the reply was lost
retrying charges the customer twice
A blind retry is a coin flip between those two. Retry is not a second chance, it is a second execution.
FigureA timeout does not mean it did not happen. It means you have no idea whether it happened, and no cleverness on your side can recover the difference.

That is the line worth keeping: retry is not a second chance, it is a second execution.

The four doors a duplicate comes through

1 Retry
client or SDK
2 Redelivery
queue or webhook
3 Redispatch
expired lease
4 Double trigger
input side
The failure point
The gate in front of the side effect is missing, so every door becomes a double charge.
Add one idempotency key there and every door is caught, regardless of which one fired. Retry is a second execution, not a second chance.
FigureA duplicate can enter through any of four doors, but it only becomes a double charge when there is no gate in front of the side effect.

Door one: retry

The request committed, the reply got lost, and the client fires the same action again.

Two things make this sneakier than it sounds. Most SDKs retry on their own by default, often more than once, so you may be retrying without ever having written a retry. And when they retry, they rerun the whole step, side effect included, because the SDK has no idea which parts of your step were effectful.

The detection signature: the same logical action, the same identifier if you have one, milliseconds to a couple of seconds apart, with the first attempt showing a timeout or a connection reset rather than an error response.

This is TCP resending a segment it got no acknowledgement for. The difference is that every TCP byte carries a sequence number, so the receiver sees the duplicate and drops it. The sequence number is an idempotency key, one layer down.

Door two: redelivery

This one is not even a bug. It is the guarantee doing exactly what it promised.

A queue or a webhook that has been told "if you are not certain you delivered it, deliver again" will sometimes hand you the same message twice. Many of them will tell you so directly: the receive count on the message literally says this is the second delivery.

This is at-least-once delivery, and it is what you signed up for when you chose the transport. The problem is not the redelivery. The problem is a handler that is not safe to run twice.

The detection signature: the same message identifier, a receive count above one, and a gap that matches your visibility timeout or your webhook retry schedule rather than anything in your own code.

HTTP bakes this distinction into its method semantics. GET, PUT, and DELETE are defined to be safe to repeat. POST is not. That is precisely why a charge needs a key and a read does not.

The mental model is a coat check. You hand over your coat and get ticket 47. Handing back ticket 47 gets you your coat. It does not get you a second coat, no matter how many times you present it.

Door three: redispatch

This is the one that wrecks agent systems most often, and it is not a retry at all.

A tool call takes a couple of minutes. Worker A is running it and will finish fine. But a sweeper decides worker A must be dead, because its lease expired or its heartbeat went quiet, and redispatches the same step to worker B.

The original did not die. It was just slow. Now two workers are running the same step, and both of them finish.

Nobody retried anything. A timer expired underneath a slow step.

The detection signature: two executions separated by exactly your lease duration or heartbeat timeout, on different workers, with the first one completing successfully after the second one started. That timing coincidence is the tell, and once you notice that the gap equals a configured timeout, the diagnosis is done.

This one gets worse as your steps get slower, which means it tends to appear in production and not in testing, because production has the slow path and the contended database and the cold cache.

At the hardware level this is a bus command resent on error, and whether it is safe depends entirely on what you are writing. Set the thermostat to 72 twice and nothing happens. Turn it up by two degrees twice and you are sweating. Absolute writes are safe to repeat. Increments double-count.

That distinction is worth carrying into your own design. Wherever you can express an action as "make the world be in this state" rather than "change the world by this amount," repetition stops mattering.

Door four: double trigger

The simplest of the four. Two copies enter with different identifiers, milliseconds apart. Either a framework double-invoked the tool, or the model re-emitted the same call in its loop with a fresh identifier, and nothing downstream recognizes the second as a repeat.

The detection signature: different identifiers, close together, with separate traces, which distinguishes it from the first three doors immediately.

Your database will reject this for you, but only if you set it up to. A unique constraint on the natural key of the action, plus an insert that does nothing on conflict, means the second insert quietly does nothing. The important property is that the database decides the winner in one atomic step.

A double needs two things: a duplicate and a missing gate

Here is the hinge of the whole thing, and it is why you usually do not need to know which door you have.

Look at the trace immediately before the side effect fires. The agent reports success. The model intended one call. The tool ran, then ran again. And nothing anywhere compares the one intended against the two executed.

No gate at the boundary. No reconciliation afterward.

All four doors are extremely common. Duplicates are normal, expected, and unavoidable in any distributed system. The missing gate is what makes them dangerous.

Put a gate there and a duplicate is a non-event: it gets recognized, the first result gets returned, nothing happens twice. Leave it out and every one of the four doors becomes a double charge.

NetworkTCP sequence numbers let the receiver drop a duplicate segment
ProtocolGET, PUT and DELETE are defined safe to repeat. POST is not
Queueat-least-once delivery is the guarantee, not the bug
Databasea unique constraint plus insert-on-conflict decides the winner atomically
Fleetdesired state, so the second apply is a no-op rather than a second change
Your agent's side effectusually the one layer with no gate at all
Every side effect is either idempotent or a bug waiting for a retry. There is no third category.
FigureThe same question appears at every layer of the stack, long before there was a model in the system. Whoever answers it correctly at their layer makes duplicates a non-event.

There is an entire layer of computing that made this gap impossible by design, with no model in it anywhere. A fleet system driven by desired state, where a directive that says "drive this machine to this configuration" changes nothing on the second application, because the machine is already there. The second apply is not rejected, it is simply a no-op, which is a stronger property than rejection.

The same question shows up at every layer: is this safe to run twice?

The fix is the same no matter which door

Go to the point immediately before your side effect and ask one question. Is there an idempotency key here? Yes or no.

In the overwhelming majority of agent systems I have opened, the answer was no. Nobody put one there, because nobody had a reason to until the first double charge.

The moment you learn that, stop diagnosing and add a durable gate. You do not need to know which door it came through to close all four.

If you do want to know which door

Count what the model intended against what actually ran. Those two numbers should always match, and the mismatch is your double. That check does double duty, because leaving it running permanently is half the fix: it is the reconciliation half.

Then look at identifiers and time gaps, which separate the doors cleanly. Same identifier, milliseconds apart, first attempt timed out: retry. Same message identifier with a receive count above one: redelivery. Two runs separated by exactly your lease duration, on different workers: redispatch. Different identifiers, seconds apart, separate traces: double trigger.

  1. Same identifier, milliseconds apart, first attempt timed out
    door one, a client or SDK retry
  2. Same message identifier, receive count above one
    door two, at-least-once redelivery doing what it promised
  3. Two runs separated by exactly your lease duration, different workers
    door three, redispatch under a slow step. The gap equalling a configured timeout is the tell
  4. Different identifiers, seconds apart, separate traces
    door four, something upstream fired twice
Count what was intended against what ran first. The mismatch is your double, and leaving that check running is half the fix.
FigureYou rarely need to know which door a duplicate came through, because the fix is the same. If you do want to know, identifiers and time gaps separate them cleanly.

You never touch the model to do any of this.

The per-door fixes

Retry. Attach a stable idempotency key, generated once when the action is first conceived and reused across every attempt. This is the single most common way teams get it wrong: generating a fresh random key per attempt defeats the entire mechanism, because every retry then looks like a new action. The key belongs to the intent, not to the attempt.

Redelivery. Deduplicate on the message identifier, or make the write an upsert so a second application changes nothing.

Redispatch. Move the step onto durable execution, so finished steps are skipped on resume, and make the step itself idempotent anyway. Be honest with yourself that durable execution is not the same as exactly-once, because nothing is. It reduces the window, it does not eliminate it.

Double trigger. Reject the second with an atomic claim: a unique constraint or a compare-and-set, in one operation. This has to be one operation. If you check whether you have seen this action and then separately record that you have, two requests can slip through the gap between the check and the write. That gap is the entire bug, and it is measured in microseconds, which is plenty.

Two rules that sit above all four

Classify the error before you retry. A clear, definite failure is safe to retry. An ambiguous timeout is not, because it may already have worked. Put the ambiguous case behind the key first, then retry freely.

Order your steps so the irreversible one is last. If a step must run and cannot be undone, it should be the final thing that happens, after everything that could still fail has already succeeded. You can refund a charge. You cannot un-send an email.

The lesson underneath

None of that was prompt engineering, and none of it was a smarter model. Every one of the four doors lives in the plumbing between your request and the thing that actually happens in the world.

The bigger rule is worth stating plainly: every side effect your system takes is either idempotent or it is a bug waiting for a retry. There is no third category. A side effect that has never been duplicated has not been proven safe, it has been lucky.

And "is this safe to run twice" is the same question at every layer, from the network to the protocol to the queue to the database and out to the fleet. It is the same discipline whether there is a model in the system or not.

FAQ

Why does my AI agent charge a customer or run an action twice? Because the action was correct and ran twice, not because the model erred. A duplicate enters through one of four doors, a client or SDK retry, at-least-once redelivery from a queue or webhook, a redispatch after a lease expired under a slow step, or a double trigger, and it hurts you only when there is no gate in front of the side effect.

Will a better model stop an agent from double-executing? No. In every case the model already emitted the correct action, and the duplication happens downstream in the plumbing. The fix is an idempotency key and a gate before the side effect, which the model has no effect on.

What is an idempotency key and why does it fix double execution? It is a stable identifier attached to an action so the server can recognize a repeat and return the first result instead of running it again, like a coat-check ticket. The key must be generated once when the action is conceived and reused across all attempts, because a fresh key per attempt makes every retry look like a new action.

Is a timeout safe to retry? No, a timeout is ambiguous, it may mean the action succeeded and only the reply was lost. Blindly retrying is a coin flip between saving the day and double-executing. Put the action behind an idempotency key first, then the retry is safe.

Why do two workers run the same agent step? A lease or heartbeat expired underneath a slow step, so a sweeper concluded the worker was dead and redispatched the work. The original was not dead, only slow. The tell is two executions separated by exactly your lease duration, on different workers, with the first still completing successfully.

Does durable execution give me exactly-once? No. It skips completed steps on resume, which narrows the window considerably, but nothing provides true exactly-once across a network boundary. Make the step idempotent anyway, and put the irreversible action last.

Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program