Desired State: How Fleet Systems Made Double Execution Impossible

Mostafa Dhouib·September 7, 2026·7 min read

Desired State: How Fleet Systems Made Double Execution Impossible

The short answer. An imperative instruction repeated twice happens twice. A declared desired state applied twice happens once, because the second application observes the world already matches and does nothing. Fleet infrastructure adopted this because at scale retries are not an edge case, they are continuous. Three properties make it work: the intent describes a state rather than an action, the system observes actual state before acting, and convergence is idempotent. Borrowing the pattern without all three is where it stops helping.

Anyone building agents that take actions is currently working through a problem infrastructure engineering resolved a decade and a half ago. It is worth taking the answer rather than rediscovering it, and worth understanding precisely why it works, because the shallow version does not.

The problem, in one line

You send an instruction. The network drops the acknowledgement. You do not know whether it ran.

Retry and you might do it twice. Do not retry and you might not have done it at all. At a hundred nodes this is a rare annoyance. At twenty-five thousand nodes it is happening continuously, to somebody, all the time.

Notice that no component here is broken. The instruction was correct, the node executed correctly, and the acknowledgement was lost by a network doing what networks do. The ambiguity is structural, and no amount of correctness in the parts removes it.

The move: describe the state, not the action

Imperative: "Set the power cap to 250 watts." Applied twice, it runs twice. If the operation is additive or has side effects, twice is wrong.

Declarative: "The power cap should be 250 watts." Applied twice, the second application reads the current cap, sees 250, and does nothing.

That is the whole idea, and stated that plainly it sounds too simple to carry weight. It carries weight because of what it changes about the retry decision: you no longer have to know whether the first attempt succeeded. The ambiguity that made retry dangerous stops mattering, because the second attempt is safe regardless of the answer.

The three properties that actually do the work

This is the part that gets lost when the pattern is borrowed, and each of the three is a place people stop early.

One: the intent is a state, not an action. "Reboot the node" is an action and it does not have this property, because rebooting twice is two reboots. "The node should have been up since after time T" is a state, and it can be evaluated. Some operations resist this conversion, which is a real limit rather than a failure of imagination, and the honest response is to name those explicitly rather than to pretend the pattern covers them.

Two: the system observes actual state before acting. This is the property most often missing. A system that records "I set the cap to 250" and skips the operation next time is trusting its own memory, and its memory is exactly what the lost acknowledgement made unreliable. It has to go and read the current value from the node. Observation is what makes convergence real rather than bookkeeping.

Three: convergence is idempotent, including the observation path. Applying the desired state when the world already matches must be a genuine no-op, and reading state must not itself change anything. This sounds obvious and breaks in practice on operations that reset a counter when read, or on writes that touch a modification timestamp even when the value is unchanged.

Where all three hold, retry is free. Where any one is missing, you have a declarative interface over an imperative system, which is worse than an honest imperative one because it invites unlimited retries into something that cannot take them.

What this buys at fleet scale

Retries become uninteresting. The most valuable property, because it removes an entire category of design decision. You can retry on any failure, at any time, from any component, without a coordination protocol.

Convergence loops become safe. A controller can continuously reapply desired state, which means drift gets corrected without anyone noticing there was drift. Somebody changing a setting by hand on a node gets quietly reverted.

Partial failure stops being special. Half the fleet applied and half did not is not a recovery scenario. Run it again.

The audit story improves. The record of what should be true is separate from the record of attempts, so "what is this node supposed to look like" has an answer that does not require replaying a log.

What it does not solve

Worth being precise, because the pattern gets oversold and then distrusted.

Genuinely non-idempotent actions. Sending an email, charging a card, launching a physical process. These have no state formulation, and the answer there is a different mechanism: an idempotency key carried by the caller, deduplicated at the boundary that performs the action.

Ordering. Desired state says what should be true, not what should happen first. Where order matters, you still need it expressed somewhere.

The gap between declared and achievable. A state that cannot be reached produces a system that tries forever, quietly. Convergence loops need their own progress measure and their own giving-up state, or they become the infrastructure version of an agent that will not stop.

That last one is a real and common failure: a reconciliation loop that has been trying to reach an impossible state for three weeks looks identical, from the outside, to one that is working.

Why this matters for agents

An agent taking actions is in exactly the position of a fleet control plane: it issues operations over an unreliable path, it retries, and it frequently cannot tell whether the previous attempt landed.

The translation is direct.

Prefer tools that express state over tools that express actions. "Ensure this record has these values" rather than "update this record." The tool's contract, not the prompt, is where this is decided.

Have the tool read before it writes. Then a repeated call is observably a no-op, and the agent's tendency to retry stops being a liability.

For the actions that cannot be expressed as state, use an idempotency key generated from the task rather than from the attempt, deduplicated at the service boundary. This is the mechanism payment systems have used for years, and it is available to anyone.

Give convergence loops a residual and a give-up state. The same requirement as any other loop: a measure of remaining work that shrinks, and a distinct outcome when it stops shrinking.

Infrastructure got here because at fleet scale the alternative was unmanageable. Agent systems are arriving at the same constraint from a different direction, and the answer has been sitting in a different discipline the whole time.

FAQ

What is desired state configuration? Declaring the state a system should be in rather than the action to perform. Applied twice, the second application observes the world already matches and does nothing, which makes retry safe without knowing whether the first attempt succeeded.

Why does declarative infrastructure prevent double execution? Because the ambiguity that makes retry dangerous, not knowing whether the first attempt landed, stops mattering. The second attempt is a no-op if the first succeeded and completes the work if it did not, so you never need to resolve the ambiguity.

What makes desired state actually work? Three properties together: the intent describes a state rather than an action, the system reads actual state from the target before acting rather than trusting its own record, and convergence is genuinely idempotent including the observation path. Missing any one gives you a declarative interface over an imperative system, which is worse than an honest imperative one.

What can't be expressed as desired state? Genuinely non-idempotent actions such as sending an email, charging a card, or launching a physical process. Those need an idempotency key generated from the task and deduplicated at the boundary that performs the action, which is the mechanism payment systems use.

How does this apply to AI agents? An agent issuing tool calls is in a control plane's position: unreliable path, frequent retries, and no way to tell whether the last attempt landed. Prefer tools whose contract expresses a state, have them read before they write, and use task-derived idempotency keys for the actions that cannot be expressed that way.