When Not to Ship an Autonomous Agent

Mostafa Dhouib·September 6, 2026·7 min read

When Not to Ship an Autonomous Agent

The short answer. One test decides it. Can you compute, in code and without asking the model, a number that shrinks as the task gets closer to done? If yes, an autonomous agent is buildable and the number is what makes it reliable. If no, and you cannot manufacture one by restructuring the task, then ship a bounded assistant with a human terminal state instead. That is not a downgrade. It is the same capability with the failure mode taken out.

Almost everything written about agents is about how to build them. This is about when not to, which is a shorter piece and a more useful one, because the decision is binary and the test takes an afternoon.

The test

Write down the task. Then answer one question:

Is there a value, computed in code from ground truth, that decreases as the work gets closer to finished?

Not a value the model reports. Not a confidence score. A number your own code can produce by looking at the world, which is smaller when more of the job is done and reaches zero when it is complete.

For some tasks this is trivially available:

Migrate these 40,000 records. Rows not yet migrated. Counted in the database, no model involved.

Fix this failing test suite. Tests still failing. The test runner produces it.

Reconcile these invoices against the ledger. Unmatched lines. Arithmetic.

Fill these fields from these documents. Fields still empty, cross-checked against the source.

For other tasks it does not exist, and no amount of engineering conjures it:

Research this market and write a strategy memo. What decreases? Sections written is a proxy for effort, not progress, and a memo can get worse as it gets longer.

Investigate this outage and find the root cause. Hypotheses eliminated sounds like a residual until you notice the space is not enumerable.

Improve this codebase. Improve toward what, measured how, and finished when?

The pattern is that a computable residual requires the finish line to be a fact about the world rather than a judgement about quality. Where the finish line is a judgement, only a judge can call it, and the model is not a judge because it is not consistent with itself between runs.

Why this is the deciding factor

The reason this single test carries so much weight is that without a residual, an agent has no way to stop for the right reason.

A loop terminates provably when there is a value that strictly decreases and is bounded below. If your only decreasing value is an iteration counter, the loop does terminate, and it terminates because the budget ran out rather than because the work finished. Those are different events and most systems return the same thing for both.

So an agent on a task with no computable residual has exactly two possible endings: the model says it is done, which is a sample from a distribution and not reproducible, or the counter hits zero, which means nothing about the outcome. Neither is a completion signal. Both are reported as one.

That is not a quality problem you can improve your way out of. It is structural, and it is why these systems demo well and fail in production: a demo is one run that happened to end well, and there was never a mechanism that would have made it end badly in a visible way.

Before you give up: three ways to manufacture a residual

Most tasks that look residual-free are actually a bundle. Unbundling them is usually the highest-value hour in the design.

Decompose until the leaves are checkable. "Investigate this outage" has no residual. "Check these fourteen hypotheses against these logs, each with a pass or fail" has one: hypotheses unchecked. The judgement moved up to a human choosing the fourteen, and the mechanical work below it became boundable.

Convert quality into coverage. "Write a good strategy memo" has no residual. "Answer these nine specific questions, each with a cited source" has one: questions unanswered. You have not measured quality, but you have made incompleteness detectable, which is most of what you needed.

Add a checkable artifact. "Improve this codebase" has no residual. "Make these twelve new tests pass without breaking the existing suite" has one, and it is produced by a tool that has no opinion.

The common move in all three: push the judgement to a human at the boundary, and let the agent operate only where a machine can tell whether it is finished.

What to ship instead

When you genuinely cannot manufacture a residual, the answer is not a worse agent with a bigger prompt. It is a different product shape.

A bounded assistant with a human terminal state. The system does the work, and the run ends by presenting a result to a person rather than by acting on it. The stopping decision is a human decision, which is fine: humans are the correct instrument for judgement calls, and this design uses them for precisely that and nothing else.

This is worth defending, because it is often treated as the timid option. It is not. Compare the two honestly:

The capability is identical. The difference is entirely in who calls the finish, and in the residual-free case only one of them can call it correctly.

The version worth building later

This is not permanent. A task without a residual today often has one after you have run the assistant for a few months, because you accumulate the thing you were missing: a record of what a correct output looked like, and reviewer decisions attached to real cases.

That is a dataset you can turn into a check. Once a check exists, the residual exists, and the autonomous version becomes buildable for real rather than aspirationally.

The sequence that works is: bounded assistant first, gather the judgements, build the check from them, then remove the human from the cases the check covers and leave them on the cases it does not. What does not work is starting at the end and hoping the prompt holds.

FAQ

When should I not build an AI agent? When you cannot compute, in code and without asking the model, a number that shrinks as the task gets closer to done. Without that number the loop can only stop because the model said so or because the budget ran out, and neither is a completion signal.

What is a computable residual? A value your own code produces by looking at ground truth, which decreases as work is completed and reaches zero at the end: rows unmigrated, tests failing, invoice lines unmatched, fields still empty. It requires the finish line to be a fact about the world rather than a judgement about quality.

My task has no obvious residual. Is that the end of it? Usually not. Most residual-free tasks are bundles. Decompose until the leaves are checkable, convert quality targets into coverage targets, or add an artifact a tool can check. The move is always to push judgement to a human at the boundary and let the agent work only where a machine can tell it is finished.

Is a bounded assistant a lesser product? No. The same work gets done; the difference is who calls the finish. In the residual-free case a person is the only party who can call it correctly, so the assistant is the design that matches the problem rather than a compromise on it.

Can a task become agent-ready later? Yes, and that is the normal path. Running a bounded assistant accumulates reviewer judgements on real cases, which is exactly the raw material for a check. Once the check exists the residual exists, and the autonomous version becomes buildable on evidence rather than on hope.