You Cannot Cancel What Does Not Heartbeat: Cooperative Cancellation And The Button That Does Nothing

Mostafa Dhouib·10 min read

You Cannot Cancel What Does Not Heartbeat: Cooperative Cancellation And The Button That Does Nothing

The short answer. In at least one widely used workflow engine, cancellation is delivered to work only through the heartbeat channel. Work that does not heartbeat never learns it has been cancelled. If you wrap a long-running command-line process, which is how most teams run an agent, and that wrapper never reports in, then pressing cancel marks the task cancelled in the orchestrator while the process runs to completion in the background, burning tokens for a result nobody will read. The control appears to function and does nothing. Cancellation is cooperative nearly everywhere, so it must be designed into the work, with a hard timeout as the only non-cooperative backstop.

The first time we saw this, the symptom was a bill. Runs were being cancelled from the operator console. The orchestrator's state said cancelled. The token spend said otherwise, and it kept climbing for several minutes after each cancellation.

Nothing in the orchestrator was broken. We had simply assumed cancellation was something done to work, and it is not. It is something work agrees to.

Why cancellation is cooperative at all

The obvious question is why an orchestrator cannot just stop the work. It is running the work, after all.

Because in the general case it is not. The orchestrator holds state about a task. The work itself runs somewhere else: a different process, a different machine, a different availability zone, inside a container that may be behind three layers of scheduling. The orchestrator's connection to it is a channel, and a channel can only deliver a message to something that is listening.

Even where the orchestrator could kill the process, killing is usually the wrong first move. Work in the middle of a transaction, a file write, or an external API call needs a chance to stop at a safe point. Terminating it abruptly produces exactly the partial state that durable execution exists to avoid. A half-written file, an API call made but not recorded, a lock held forever.

So orchestrators do the sensible thing. They set a flag and let the work notice. The work checks the flag at points where stopping is safe, then unwinds cleanly. That is cooperative cancellation, and it is the right design.

The sharp edge is in how the flag gets delivered. In the engine that caught us, the delivery mechanism is the heartbeat response. Work heartbeats to say it is still alive, and the engine answers that heartbeat with, among other things, whether the work has been cancelled. There is no separate channel. The cancellation signal rides along on the liveness signal.

Which means the two properties collapse into one. Work that does not report liveness cannot receive cancellation. Those feel like independent features and they are the same feature.

The failure is loud in the bill and silent everywhere else

Walk through what an operator sees.

They press cancel. The call returns successfully, because the engine accepted the cancellation request and recorded it. The task's state becomes cancelled. The console updates. Any dashboard reading task state shows the task as cancelled. Alerting sees a cancelled task, which is not an error, so nothing fires.

Meanwhile the child process has no idea. It never heartbeated, so it never asked, and nothing told it. It continues through its plan, makes its model calls, writes its files, and exits normally some minutes later.

Now there are two problems rather than one. The spend continues, which is the visible one. And the process writes to the workspace after the orchestrator has declared the task over, which is the dangerous one. The orchestrator may already have started a replacement run or a recovery path, and now two processes are editing the same workspace with neither aware of the other. We have seen a cancelled run and its replacement produce a merged diff that neither of them would have produced alone.

This is the same structural problem as an agent that reports success having done nothing, inverted. There, the system trusted the work's claim about itself. Here, the system trusts its own claim about the work. In both cases a state record is treated as evidence about the world.

Wrapping a process that was never designed for this

Most agent work is a command-line runner invoked as a child process. That runner was built to be used by a human in a terminal, where cancellation means pressing a key combination and the operating system sends a signal. It has no notion of an orchestrator, no heartbeat, and no cancellation flag to check.

You are responsible for bridging that gap, and the bridge has four parts.

Heartbeat from the wrapper, not the runner. The wrapper is your code. It can heartbeat on a timer while the child runs, independently of what the child is doing. This single change is what makes cancellation deliverable at all. It costs a background timer.

Check the heartbeat response for cancellation. Heartbeating and ignoring the answer is a common half-measure. It fixes the liveness problem and leaves the cancellation problem exactly where it was.

Translate cancellation into a signal the child understands. On receiving cancellation, send the child the termination signal it would get from a terminal. A well-behaved process cleans up and exits. This is where cooperation ends and the operating system begins.

Kill after a bounded grace period. A process that ignores the polite signal gets killed outright. This is the part teams leave out, and it is the only part that is not cooperative.

The order matters, and so does the fact that every step has a deadline. A wrapper that sends a termination signal and then waits indefinitely for the child to exit has reintroduced the original bug one layer down. The child is now the thing that must cooperate, and it may not.

Timeout as the last resort, not the first line

A hard timeout is the only cancellation mechanism that does not require the work's participation. It is also the bluntest, which is why it belongs at the bottom of the stack rather than the top.

The pattern we use has three layers and each one is strictly less polite than the one above it.

Cooperative cancellation, immediately. The work is told to stop and stops at a safe point. This is the only layer that produces clean state, so everything else is a degradation.

Signal escalation, after a grace period measured in seconds. The child is sent a termination signal. The grace period should be long enough for a genuine cleanup and short enough that an operator does not conclude the button is broken. We use single-digit seconds for agent runners, because their cleanup is small.

Hard kill and a hard timeout, as the floor. The process is killed. Separately, and independently of any cancellation, every run has a wall-clock timeout after which it is killed whether anyone asked or not. That timeout exists for the case where the wrapper itself has died or wedged, so it cannot live in the wrapper. It lives in the orchestrator or in the platform.

The crucial design consequence is that layers two and three can leave the workspace in a partial state. That is acceptable only if something downstream treats the workspace as untrusted after a kill. In practice that means a killed run's workspace is discarded and rebuilt rather than resumed, and the task is marked for a fresh start rather than a continuation. If you skip that, you have traded a run that would not stop for a run that stopped halfway and got built on.

What to verify before you believe your cancel button

The drill is three steps and it is the only way to know.

Start a long-running task through your real wrapper, not a test stub. Press cancel through the real operator path. Then, separately from any orchestrator state, confirm the process is gone and the spend has stopped. Check the process table. Check the provider's usage, not your own counter, because your counter is part of the system under test.

If the task shows cancelled and the process is alive, you have found it. If the process dies but the spend continues for a while, you have found a different version, where the child was killed but an in-flight request was not.

Record the time from pressing cancel to the process actually being gone. That number is the real behaviour of your cancel control, and it is usually the first time anyone has measured it.

The honest limit

The heartbeat-delivery behaviour is specific. It is documented behaviour of at least one widely used workflow engine, and other engines deliver cancellation differently, some through a separate channel that does not depend on liveness. Do not generalise the mechanism without checking your own engine's documentation, and do not assume your engine is safe because this one is not.

Our reproduction was a lab exercise on a wrapped runner we built to be deliberately uncooperative, not a client postmortem, and we are not putting a figure on how much spend this leaks in practice. That figure depends entirely on how long your runs are and how often anyone cancels.

The layered pattern has a real cost that is easy to undersell. Heartbeating from the wrapper means the wrapper now claims the work is alive, which is true, but says nothing about whether the work is making progress. A wedged child process with a cheerfully heartbeating parent is a new failure mode you did not have before, and detecting it needs a progress signal rather than a liveness signal. Fixing cancellation does not give you stall detection, and the two are easy to confuse because both are about a run that will not end.

FAQ

Why does my cancel button not stop the agent run? Most likely because cancellation is cooperative and your work is not participating. In at least one widely used workflow engine, cancellation is delivered only through the heartbeat response, so work that never heartbeats never learns it was cancelled. The engine accepts the request and marks the task cancelled while the underlying process runs on, which is why the control looks like it worked.

Why is cancellation cooperative instead of just killing the process? Because the orchestrator usually does not hold the process, only a record of the task, and because killing work mid-transaction produces the partial state that durable execution exists to prevent. A half-written file or an API call made but not recorded is often worse than a run that takes longer to stop. Cooperative cancellation lets work stop at a point where stopping is safe.

How do I make a wrapped command-line process cancellable? Heartbeat from your wrapper on a timer rather than from the child, read the cancellation state out of the heartbeat response, translate it into the termination signal the child would receive in a terminal, and kill the child outright after a bounded grace period. All four parts are needed. Heartbeating without reading the response fixes liveness and leaves cancellation broken.

Where should a hard timeout live? Not in the wrapper, because the wrapper is one of the things that can wedge. Put the wall-clock timeout in the orchestrator or the platform, so it fires whether or not your code is healthy. Treat it as the floor beneath cooperative cancellation and signal escalation, not as the primary mechanism, because a timeout can leave the workspace in a partial state.

How do I test whether cancellation actually works? Start a long task through the real wrapper, cancel it through the real operator path, then verify outside the orchestrator that the process is gone and the spend has stopped. Check the process table and the provider's usage rather than your own counters, since your counters are part of the system under test. Record how long the whole thing took.

What happens to the workspace after a hard kill? Assume it is partial and untrusted. A killed run may have written half a file or made an external call it never recorded, so a resume built on that workspace is building on a state nobody designed. Discard it and start the task fresh. Skipping this turns a run that would not stop into a run that stopped halfway and was then treated as a foundation.