Six Drills That Break An Agent: Accept Milestones By Breaking Them, Not By Demonstration

Mostafa Dhouib·10 min read

Six Drills That Break An Agent: Accept Milestones By Breaking Them, Not By Demonstration

The short answer. A milestone on an agent system should be accepted by drills that break it on purpose, not by a demonstration that it works. A demonstration is run by the person who built the thing, along the path they built it for, and it proves only that the happy path exists. Six drills cover most of what actually goes wrong in production: kill the worker mid-run, hit the output token cap, resume a run and watch the budget, cancel a run that does not heartbeat, rate-limit mid-run, and make a change through the shell that the tool's checkpointing never saw. Each one has a correct answer and a common failure, and the common failure is usually silent.

Demonstrations are not useless. They are just a different instrument. A demonstration answers "can this work," which is a question you usually already know the answer to by the time you are scheduling a demonstration.

Acceptance needs to answer a harder question: what does this system do when something goes wrong that nobody wrote code for. You cannot get that from a demonstration, because a demonstration is a scripted run and the script does not include the failure.

So we accept agent milestones with drills. A drill is an adversarial procedure with a stated expected behaviour, run by someone who did not write the code, with the result written down. Here are the six we run first.

Drill one: kill the worker mid-run

The procedure. Start a job that takes longer than your queue's visibility or lease window. Kill the worker process without warning, not with a graceful shutdown signal. Watch the queue.

What a correct system does. The job either comes back to be retried by another worker, with the previous partial work either rolled back or safe to repeat, or it is parked for a human with a clear reason. Both are acceptable. What matters is that the system's behaviour is the one the team expected.

The common failure. Two opposite ones, depending on the queue. Either the job is handed to a second worker while the first is still alive and merely slow, and now the same work runs twice, or the dead worker's lease is held for hours and the job simply stalls with nothing to show for it. Teams usually discover which of these they have during an incident, because the happy path in a job queue is genuinely easy and hides the question for months.

The drill takes ten minutes and it tells you which half of the problem you own.

Drill two: hit the output token cap

The procedure. Give the agent a task whose honest answer exceeds the maximum output tokens configured for the call. A large file rewrite, a long structured document, a full report. Do not warn it. Let it run into the ceiling.

What a correct system does. It detects that generation stopped because of the cap rather than because the model finished, treats that as an incomplete result, and either continues from where it stopped or fails loudly. The distinguishing signal exists in the response, and reading it is a one-line check.

The common failure. The truncated output is accepted as a complete answer. This is at its worst with structured output, because a truncated object is invalid and a truncated code file is syntactically broken, which at least fails fast. The dangerous case is truncated prose or a truncated list, which parses fine and is simply missing the end. A pipeline that writes that to disk has recorded a wrong answer with no error anywhere.

Watch for the version of this where a retry on a parse error eventually produces a shorter, lower-quality answer that fits. The system then looks like it recovered when what it actually did was degrade.

Drill three: resume a run and watch the budget

The procedure. Set a spend or token budget on a run. Let the run consume most of it. Interrupt the run. Resume it. Now watch the counter.

What a correct system does. The budget is a property of the task, held outside the process, and resuming continues from the consumed total. A run that had five per cent of its budget left before the interruption has five per cent of its budget left after.

The common failure. The counter is held in the process, so it dies with the process and starts again from zero on resume. A run with a cap can then spend several times the cap by being interrupted and resumed repeatedly, and every individual run stays inside the limit. The audit trail shows compliance. The invoice does not.

This one is worth drilling specifically because interruption and resume are not rare in agent systems. They are the normal response to almost every other failure in this list, which means the budget bug compounds with everything else.

Drill four: cancel a run that does not heartbeat

The procedure. Start a long-running agent task through whatever wrapper you use to shell out to the runner. Press cancel. Then check whether the underlying process is still alive and still spending money.

What a correct system does. Cancellation reaches the work, the child process receives a termination signal, and after a bounded grace period it is killed outright if it has not stopped. The orchestrator records the task as cancelled only after confirming the process is gone.

The common failure. In at least one widely used workflow engine, cancellation is delivered only to work that heartbeats. A wrapped command-line process that never reports in cannot be cancelled. The cancel control returns successfully, the task is marked cancelled in the orchestrator, and the process carries on to completion in the background, burning tokens for a result nobody will read. This is the single most convincing demonstration of why self-reported status is worthless, because here the orchestrator's own state is wrong.

The fix is a timeout as a last resort behind cooperative cancellation, plus an explicit process kill rather than a polite request.

Drill five: rate-limit mid-run

The procedure. Constrain the agent's quota so the run hits a rate limit partway through its task. Let it return. Then look at the workspace rather than the report.

What a correct system does. The orchestrator checks the result against the workspace before accepting it, finds no diff and no new files, and classifies the run as blocked rather than complete.

The common failure. A headless run can return a success status with exit code 0, having written nothing at all. The runner handled the rate limit as a condition rather than an error, finished its turn cleanly, and exited normally. Every downstream feature that reads the run's own account then repeats the false claim. Retry does not fire, because there was no failure. Resume finds nothing to resume. The pane goes green.

Drill six: change the workspace through the shell

The procedure. Let the agent do some work through its own editing tools. Then have it run a shell command that modifies the workspace, a formatter, a code generator, a package install, a file move. Now restore the tool's checkpoint and inspect the result.

What a correct system does. Either the restore reverts everything in the workspace, including what the shell did, or the tool refuses to restore and says plainly that it cannot guarantee the state, naming what it does not control.

The common failure. A tool that snapshots its own view of the workspace only knows about changes it made itself. Restoring reverts those and leaves the shell's changes in place, producing a state that never existed at any point in the run. That is strictly worse than having no checkpointing, because a known-dirty workspace gets rebuilt from scratch while a confidently restored one gets built on.

Running the drills so the results survive

The drills are easy. Keeping them honest is the part that needs discipline.

Write the expected behaviour before you run it. A drill without a stated expectation becomes a story about what the system did, and whatever it did will sound reasonable in hindsight. Write down the answer you expect, then find out.

Have someone else run it. The author of a component knows where not to push. This is not a comment on anyone's integrity. It is the same reason test engineers exist.

Record the result as a table of drill, expectation, observed, verdict. Six rows. That table is the acceptance artefact for the milestone, and it is more useful six months later than any demonstration recording.

Keep the ones that can be automated. Several of these become permanent checks. The diff gate from drill five runs on every milestone for free. The truncation check from drill two is a single condition in the response handler. Drills that cannot be automated, like the checkpoint one, go on a release checklist instead.

The honest limit

These six are the drills whose failures we have reproduced, and most of those reproductions were lab work on deliberately constrained runs rather than client incidents. They tell you how the system responds to a provoked condition. They do not tell you how often that condition occurs for you, and we do not have a defensible number for that.

The list is also incomplete by construction. It covers interruption, limits, cancellation and state capture, because those are where agent systems inherit assumptions from components that were not built for agents. It says nothing about the agent doing the wrong thing competently, which is a correctness problem and needs evaluation work rather than drills. It says nothing about prompt injection through retrieved content. And a system that passes all six can still be the wrong system for the task, which no drill will tell you.

Six drills passing means the failure modes you know about are handled. That is a real result and it is not the same as reliability.

FAQ

What is the difference between a demonstration and a drill? A demonstration is run by the person who built the system, along a path they chose, and it shows that the happy path exists. A drill is an adversarial procedure with an expected behaviour written down in advance, run by someone else, that breaks the system on purpose and records what happened. Acceptance needs the second one, because production supplies the failures a demonstration leaves out.

Why drill cancellation separately from failure? Because cancellation is cooperative in most orchestrators, and in at least one widely used workflow engine it is delivered only to work that heartbeats. A wrapped command-line process that never reports in cannot be cancelled, so the control returns successfully and the process keeps running and spending. Nothing about a crash or an error drill will surface that, because nothing crashed.

Which drill should I run first? Rate-limit mid-run, because it is the cheapest and it tests your acceptance logic rather than a component. A headless run can come back with a success status and exit code 0 having written nothing, so if your pipeline accepts that run, every other drill result is unreliable too. Fix the acceptance gate first, then drill the components behind it.

Can these drills be automated into continuous integration? Partly. The output truncation check and the empty diff gate become permanent conditions in the pipeline and cost almost nothing. Worker kills and rate limits can be automated with effort, usually with a fake provider. Checkpoint restore after a shell change is difficult to automate honestly and is better kept as a release checklist item run by hand.

How long should six drills take? Each individual drill is minutes of execution. The cost is in deciding the expected behaviour beforehand and in building the harness that lets you provoke a rate limit or kill a worker on demand. That harness is reusable, so the first pass is a day or two of work and each subsequent milestone is an afternoon.

What do these drills not catch? Correctness. Every drill here tests how the system behaves when infrastructure misbehaves, not whether the agent did the right work. An agent that confidently produces a wrong but well-formed change passes all six. They also do not cover prompt injection through retrieved content, and they say nothing about whether the task should have been given to an agent in the first place.