Agents Take Actions That Can't Be Undone: Designing for Irreversibility

Mostafa Dhouib·September 4, 2026·7 min read

Agents Take Actions That Can't Be Undone: Designing for Irreversibility

The short answer. The line between a chatbot and an agent is that an agent acts, and actions only turn one way. It sends the email, charges the card, deletes the record, places the order, at machine speed, faster than anyone can reach in and stop it. In 2012 an automated trading system shipped a bad deployment with no kill switch and lost 440 million dollars in roughly five minutes, effectively ending a company that had been around for years. The code was the last domino. The failure was the system that let it run with no brakes.

Most of the reliability conversation around agents is about whether the reasoning is good. That is the less important question.

The important one is what stands between the reasoning and the effect, because the reasoning will eventually be wrong, and everything after that depends on what you built in between.

The line that separates a chatbot from an agent

A chatbot writes words. If it is wrong, you read it, notice, and delete it. The cost is approximately zero, as long as you did not act on it.

An agent acts. And the actions available to it are not symmetric with respect to time.

This asymmetry is the entire design problem, and most systems have never sorted their actions along it. That sort is the single cheapest large win available in agent design: enumerate every action your agent can take, put each in one of the two columns, and look at the second column. Teams are consistently surprised by what is on it.

The sort is also less obvious than it first appears. Some actions are technically reversible and practically not: you can delete a message you sent, after someone read it. Some are reversible only within a window. Some are reversible in the system but not in the world, which is where a refund differs from an un-sent email. Treat "reversible" as meaning you can restore the prior state without anyone outside the system noticing, and the column gets shorter and more honest.

Machine speed is the multiplier

An irreversible action taken once by a person is a mistake. The same action taken by a loop is a category of event with no natural bound.

The 2012 trading failure is the canonical case: a bad deployment, no kill switch, and 440 million dollars gone in about five minutes. What makes it instructive is not the size but the shape. The code did what code does, which is execute. The system had no way to notice or stop it. The loss rate was faster than any human response cycle.

That is not a code problem. It is a systems problem, and the distinction matters because the fix lives somewhere the code review would never have looked.

Four layers between reasoning and effect

Preconditions. Assert the state the action assumes. An action premised on a record that no longer exists should fail before it fires rather than after. Cheap, and it catches the class where the world moved between the decision and the execution.

A dry run with limits. Compute what would change and check it against bounds before doing it. This is how you catch the transfer that is correct in form and three orders of magnitude wrong in amount, which is a failure no amount of reasoning quality prevents, because the reasoning produced a well-formed instruction.

An idempotency key in front of the side effect. A retry is a second execution, not a second chance. Without a key, every retry, redelivery, redispatch, and double trigger becomes a second real-world action.

An action class policy. Reversible actions proceed. Irreversible actions are gated, either behind a human or behind a stronger check. These should never share a code path, because sharing one means the weakest requirement governs both.

Ordering, budgets, and the stop

Three more controls that operate on the whole run rather than on a single action.

Put the irreversible step last. If a step must run and cannot be undone, it should be the final thing that happens, after everything that could still fail has already succeeded. This is not always possible, and where it is, it converts a partial failure from a mess into a no-op. You can refund a charge. You cannot un-send an email.

Bound the blast radius, not just the action. A per-action check does not stop a loop from taking a thousand individually valid actions. Rate limits on effectful operations, a cap on the number of irreversible actions per task, and a spend budget enforced outside the agent are what bound the aggregate. Every one of these is the same lesson as bounding a system of loops rather than a single loop.

Build the stop before you need it. A kill switch that has never been exercised is a hypothesis. It needs to be reachable by someone who is not the person who deployed the system, to work when the system is in its worst state rather than its normal one, and to be tested on a schedule. The reason to test it is the same reason a test that has never failed proves nothing: an untriggered safety mechanism is an assumption wearing a control's uniform.

Where the human goes

Human in the loop is often proposed as the answer and then implemented in the one position where it does the least good.

A human reviewing output after the action has fired is not a control, it is a log. A human approving every action is a control that gets removed within a month, because it does not survive contact with volume.

The version that works is a human on the gated class only, with the information needed to decide in front of them: what the agent proposes, what state it assumes, what will change, and what it was uncertain about. Reversible actions flow. Irreversible ones queue. The queue must be small enough that people actually work it, which is a design constraint on how you draw the line, not an operational detail.

And where the deadline is too short for a person, the fallback becomes a safe state rather than a human. That is a legitimate answer. What is not legitimate is having neither.

The question to ask of your own system

Take the trace of a run immediately before your most consequential side effect and ask: what would have to be true for this to fire when it should not, and what in the system would notice?

If the honest answer is that nothing would notice, the reasoning quality of your agent is not the thing to work on this quarter.

FAQ

How do I make an AI agent safe for real actions? Sort every action into reversible and irreversible, then put four layers between the reasoning and the irreversible ones: preconditions on the state the action assumes, a dry run checked against limits, an idempotency key so a retry cannot double-execute, and a policy that gates the irreversible class rather than sharing a code path with the reversible one.

Does an AI agent need a kill switch? Yes, and it needs to have been exercised. An untested kill switch is an assumption, not a control. It must be reachable by someone other than whoever deployed the system, work when the system is in its worst state, and be tested on a schedule.

Where should a human sit in an agent's loop? On the gated class only, with the proposal, the assumed state, the predicted change, and the uncertainty in front of them. Reviewing output after the action has fired is a log, not a control, and approving every action does not survive contact with volume. Where the deadline is too short for a person, the fallback is a safe state.

Why does the order of steps matter for irreversible actions? Because an irreversible step placed last runs only after everything that could still fail has already succeeded, which turns a partial failure into a no-op rather than a mess. You can refund a charge; you cannot un-send an email.

What is the difference between a chatbot failure and an agent failure? A chatbot writes words, so a wrong output costs you the time to notice and delete it. An agent acts, and its actions turn one way at machine speed, so a wrong output becomes a real-world consequence faster than anyone can intervene.