Agent ReliabilityField engagement

An agent that stops is not the same as an agent that finished, and most systems cannot tell the difference.

Fractional CTO and lead engineer.
at-least-once
made safe
completion
distinguished from stopping
050%100%15101599%/step95%/step90%/step95%/step is a coin flip at 10steps in the chainend-to-end reliability
End-to-end reliability is per-step reliability raised to the number of steps. A near-perfect step becomes a coin flip over a chain.
What was at stake

These are old distributed-systems problems arriving in new clothing, and they are the difference between an agentic system that can be trusted with real work and one that cannot. Both failure modes here are silent, and both produce results that look like success.

The constraint

Two specific problems. An iteration cap is a circuit breaker, not a completion strategy: a system that stops at step twenty reports the same shape of result whether it finished the task or ran out of budget, and nothing downstream can tell which. And a timeout on a tool call means you do not know whether the first execution succeeded, so a naive retry is a second execution rather than a second chance, which on any non-idempotent action is a duplicate side effect.

demo
123456789
production
123456X89
The demo runs clean. Production derails on one bad step, and everything after it builds on broken state.
The fork

The reflex, and the fix.

Road not taken

Cap iterations and retry on timeout

Pull

Two lines of code, and it stops the obvious runaway.

Why not

Conflates finishing with stopping, and turns every uncertain step into a possible duplicate execution.

Road taken

Explicit completion conditions and idempotency keys

Accepted

Defining what done means per task, and making every side-effecting step idempotent so a retry is safe.

Bought

A system that knows whether it finished, and one that can retry without doing the work twice.

Decision

Treat the agent as a distributed system, because that is what it is, and apply the discipline that field already has.

How it was built
01Plan
02Act
03Verify
04Complete or escalate
01

Completion as a defined condition

Each task carries an explicit success test, so finishing and running out of budget are distinguishable states with different downstream handling rather than the same shape of output.

02

Idempotency on every side-effecting step

Each action carries a key, so a retry after an uncertain timeout is recognized as the same operation rather than executed a second time. This is the standard at-least-once discipline, and it is routinely absent from agent systems.

03

Escalate rather than continue on bad state

A step that cannot be verified stops the chain and hands off, rather than passing unverified state to every step after it.

How it was measured

A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.

Task completion rate against the defined conditionDuplicate execution rateEscalation correctnessRecovery after an uncertain step
figures

What it produces
Without this discipline

A system that reports success when it ran out of iterations, and that occasionally performs the same side effect twice because a timeout was treated as a failure.

This system

An agent that knows whether it finished, retries safely because every side-effecting step is idempotent, and escalates rather than continuing on unverified state.

explicit completion conditionsidempotency keysescalate, not continue
The operating envelope

What it owns, and what it hands to a person.

Handled with confidence
Tasks with a definable success test
Actions that can carry an idempotency key
Flagged for review
Uncertain step results
Tasks approaching the budget
Out of scope by design
Tasks with no testable definition of done
Actions that cannot be made idempotent
The honest limit

Idempotency has to be designed into each action; it cannot be added generically at the orchestration layer. Tasks whose success cannot be tested cannot be given a completion condition, and those should not be automated.

What it generalizes to

Almost everything hard about agent reliability is a solved problem in distributed systems: at-least-once delivery, idempotency, circuit breaking, and explicit state machines. Recognizing which classical problem you are looking at is most of the work.

How we engage

You have a system like this one.
Tell us where it stands.

Whether it is failing, not yet built, or about to meet a scale it has never seen, we can tell you what we see.

Start a conversation
mostafa@opulion.dev · Response within 24 hours · By inquiry