Checkpoints Do Not See What The Shell Did: Partial State Capture Is Worse Than None

Mostafa Dhouib·10 min read

Checkpoints Do Not See What The Shell Did: Partial State Capture Is Worse Than None

The short answer. Checkpointing in an agent tool usually works by snapshotting the tool's own model of the workspace, which is the set of edits the tool made through its own file operations. Changes made any other way are invisible to it. Run a formatter, a code generator, a package install or a file move through the shell, then restore the checkpoint, and you get the tool's changes reverted with the shell's changes still present. That combination never existed at any moment during the run. It is worse than having no checkpoint, because a workspace everyone knows is dirty gets thrown away and rebuilt, while a workspace that was confidently restored gets built on.

The bug that taught us this was not dramatic. An agent run went wrong, we restored to a checkpoint, and the build failed with an error about a module that should not have existed yet. The module had been produced by a code generator the agent ran through the shell, three steps before the checkpoint we rolled back to. The generator's output was still there. The code that was supposed to consume it had been reverted.

Thirty minutes went into debugging a state that no sequence of operations could have produced.

What a checkpoint actually captures

The phrase "checkpoint" borrows credibility from systems where it means something strong. In a database, a checkpoint is a point in a durable log where the entire state of the system is known. In a virtual machine snapshot, it is the whole disk and memory.

In an agent tool it usually means something much narrower. The tool tracks the edits it made. It knows it wrote this content to that path, and it kept the previous content. Restoring means putting the previous content back at those paths.

That is a perfectly reasonable implementation. It is cheap, it is fast, and for the common case where the agent only edits files through the tool's own operations it is correct. The failure is not a coding error. It is a scope mismatch between what the word promises and what the mechanism covers.

The scope boundary is the tool's file operations. Inside it, the tool has a complete record. Outside it, the tool has nothing, and the outside includes:

Shell commands the agent ran. Formatters, linters with autofix, code generators, migration tools, package managers, file moves, deletions, anything invoked through a terminal.

Build artefacts and caches. Compiled output, dependency directories, lock files, generated type definitions.

State outside the filesystem. Database migrations applied, rows written, external API calls made, messages published. No filesystem snapshot was ever going to cover these, and they are the most consequential category.

Concurrent changes. Anything a human or another process did to the workspace while the run was in flight.

Why partial restore is worse than no restore

This is the part that is counterintuitive, and it is the whole point of the article.

Compare two teams. The first has no checkpointing. When a run goes wrong, they know the workspace is in an unknown state, so they discard it, clone fresh, and start over. That costs a few minutes and it is always correct.

The second has partial checkpointing. When a run goes wrong, they restore. The tool reports success. The workspace now looks plausible: files are where they should be, the structure is familiar, nothing is obviously broken. So they carry on from there.

The second team is in a worse position, and not because the restore was unsuccessful. It is worse because the restore produced confidence. A state that is known to be unknown gets handled correctly by default. A state that is believed to be known gets built on, and every subsequent action inherits the inconsistency.

This is the general shape of partial state capture, and it shows up well beyond agent tools. A backup that captures the application database but not the file store. A transaction that covers the write but not the cache invalidation. A rollback that reverts code but not the schema. In every case the dangerous property is not the gap. It is that the gap is covered by a mechanism that reports success.

The practical rule we took from this: a recovery mechanism that cannot state its own scope should be treated as having no scope.

The failure modes are not symmetric

It is worth being specific about what you actually get, because the two directions behave very differently.

Tool changes reverted, shell changes kept. This is the common case and the one described above. You end up with outputs whose inputs no longer exist. Generated code without its generator configuration. A lock file referencing a dependency that the reverted manifest no longer requests. Build failures that point at the wrong thing, because the error is in the state rather than the code.

Shell changes that destroyed what the checkpoint needed. If the agent deleted or moved a file through the shell, the checkpoint's record of that file's previous content may be gone, or the restore may recreate a file at a path where the tool expects nothing. Some restores fail here, which is the better outcome. Some succeed and produce duplicates.

Effects outside the filesystem. A migration applied cannot be un-applied by a file restore. Code gets reverted to a version that does not know about a schema change that is still present in the database. This is the version of the problem that reaches production, because it survives the workspace being rebuilt.

Notice that only the last one is visible after a clean rebuild. The first two are workspace problems and a fresh clone fixes them. That is a useful triage split.

What to verify, and how

Three things, in order of how cheap they are.

Ask the tool what its checkpoint covers, and get a straight answer. Read the documentation for the scope, not the feature description. If the documentation says it snapshots the files the tool edited, that is your scope. If it does not say, assume the narrow reading. A feature that cannot tell you its boundary does not have a trustworthy one.

Run the drill. Let the agent edit files through its own tools. Then have it run a shell command that changes the workspace in a way you can see, for example generating a file at a new path. Restore the checkpoint. Inspect the result. You are looking for whether the generated file is still there. This takes five minutes and it answers the question definitively for your version of your tool.

Check whether anything in the run reaches outside the filesystem. Migrations, external writes, published messages, cache warming. Each one is an effect no checkpoint will reverse, and each one needs its own compensating action or needs to be moved out of the agent's reach entirely.

How to work with a partial checkpoint rather than against it

We do not avoid these features. They are genuinely useful for the narrow case they cover, which is undoing an agent's own edits during an interactive session. The changes are to the surrounding process.

Treat the checkpoint as an editor undo, not a recovery mechanism. It is good for stepping back an edit you did not like while you are sitting there watching. It is not the thing you reach for after an unattended run went wrong.

Make the repository the real checkpoint. Commit before a run, on a branch. A commit captures everything in the tree regardless of which tool wrote it, which is exactly the property the tool's own snapshot lacks. Resetting to a commit and cleaning untracked files is a complete workspace restore.

Make a failed unattended run discard its workspace. Not restore it. Discard and reclone. The cost is minutes and it removes the whole class of problem. This is the single change we recommend most often and it is usually resisted, because restoring feels faster.

Push effects outside the filesystem behind an approval gate. Migrations and external writes should not happen inside an unattended agent run at all, for reasons that have nothing to do with checkpointing. Checkpointing just makes the argument concrete: anything the agent does that a reset cannot undo is permanent, and permanence deserves a human.

The honest limit

What we have described is the behaviour of tools that snapshot their own edit history, which is the common implementation but not the only one. Some tools do take a full workspace snapshot, and some use the repository underneath. If yours does, this article does not apply to it and the drill will show you that in five minutes.

Our account is a lab reproduction plus one debugging session on our own pipeline. We are not reporting a client incident and we have no figure for how often this produces a real defect, which depends entirely on how much of your agent's work goes through the shell. A run that only edits files has no exposure at all.

The repository-as-checkpoint advice has its own gap, and it is the same gap. A commit captures the tree. It does not capture a database, a message queue, a third-party system, or anything else outside the working directory. Replacing a narrow mechanism with a wider one does not get you to complete, it moves the boundary. The useful thing is knowing where the new boundary is, which is the one property the original mechanism did not give us.

FAQ

Why does restoring an agent checkpoint leave my workspace broken? Because the checkpoint captured only the edits the tool made through its own file operations. Anything the agent did through a shell command, such as running a formatter, a generator or a package install, was invisible to it. Restoring reverts the tool's edits and leaves the shell's edits in place, giving you outputs whose inputs no longer exist and a state that never occurred during the run.

Is partial checkpointing better than no checkpointing? No, and that is the surprising part. With no checkpointing, a failed run leaves a workspace everyone knows is untrustworthy, so it gets discarded and rebuilt, which is always correct. With partial checkpointing, the restore reports success and the workspace looks plausible, so people build on it. The damage comes from the confidence, not from the gap.

What should I use as a checkpoint instead? A commit on a branch, taken before the run. A commit captures the whole tree regardless of which tool wrote each file, which is the exact property a tool's own snapshot lacks. Resetting to that commit and cleaning untracked files is a complete workspace restore. For unattended runs, prefer discarding the workspace and recloning over restoring anything.

How do I test what my tool's checkpoint actually covers? Let the agent edit a file through its own tools, then have it run a shell command that creates a file at a new path, then restore the checkpoint and look. If the generated file is still present, the checkpoint covers only the tool's own operations. The drill takes about five minutes and answers the question for your exact version, which documentation often does not.

What about changes outside the filesystem? No file-level checkpoint will reverse them. A database migration applied during a run stays applied after the code that needed it has been reverted, which leaves code and schema disagreeing in a way that survives a clean reclone. Keep migrations and external writes out of unattended runs, behind an approval gate, because anything a reset cannot undo is permanent.

Can I just snapshot the whole workspace instead? You can, and it closes the shell gap, which is a real improvement. It does not make the mechanism complete. A full workspace snapshot still excludes databases, queues and third-party systems. The value of switching is not that the boundary disappears but that you know where the new boundary is, which is what the narrow mechanism never told you.