Rolling Back On A Platform That Forbids Downgrades

Mostafa Dhouib·10 min read

Rolling Back On A Platform That Forbids Downgrades

The short answer. Several connected-device platforms enforce a rule that an update must carry a strictly higher version number than the image currently running. That rule makes conventional rollback impossible, because the fix you want to ship is numbered lower than the build that broke. The technique that works is forward-versioned rollback: take the binary you know is good, relabel it with a version number higher than the bad one, and publish it as an ordinary update. The platform sees an upgrade and installs it. The device ends up running the old, safe code under a new number. The cost is bookkeeping, because version numbers no longer correspond to code, and the limit is that it only reaches devices still checking in.

A bad build is out. The old build works. Normally that is a ten minute problem.

Then the platform rejects the publish, because the version number you need to ship is lower than the one already on the devices, and the platform does not allow that under any circumstances.

Why the no-downgrade rule exists

It is not arbitrary and it is not a bug to be worked around casually. The rule exists because a downgrade is an attack.

It closes the rollback attack on security fixes. If a device will accept a lower version, an attacker who can influence what the device fetches can serve it a correctly signed older image containing a vulnerability that has since been patched. The image is genuine and the signature is valid. The monotonic version requirement is what makes a patched device stay patched.

It makes state migration one-directional. Firmware changes data formats, key material, partition layouts and configuration schemas. Upgrades migrate forward. Very little firmware implements a reverse migration, so a device moved backward may boot into code that cannot read its own stored state. Forbidding downgrades turns a class of corruption into a refusal.

It makes fleet version reasoning tractable. If version only ever increases, "every device is at or above version N" is a statement the platform can make and act on.

So the constraint is load bearing. Any technique that works around it has to preserve the properties the rule was protecting, which is the part that gets forgotten.

The technique

Forward-versioned rollback is simple enough to describe in a sentence and the discipline around it is where all the work is.

Take the binary from the last release known to be good. Do not rebuild it from source, because a rebuild is a new artefact with new risk and you would lose the one property you are relying on, which is that this exact binary has been running on the fleet without incident. Change only the version metadata so the number is above the bad build, sign it, and publish it through the normal update channel.

The platform compares numbers, sees an increase, and installs. The device ends up running proven code. The rule was satisfied on its own terms, because the version number did increase.

A few details decide whether this is a recovery or a second incident.

Leave a gap in the numbering. Do not take the next consecutive number above the bad build. Skip ahead, so there is visible space between the bad build and the relabelled good one. That gap is where the real fix will eventually live, and it makes the anomaly obvious to anyone reading the version list later.

Change only the version field. Nothing else. No opportunistic small fix bundled in, however tempting, because the entire argument for this image is that it is byte-identical to code with field history. One extra change and it is an untested build shipped during an incident.

Make the device report its true identity. The device should be able to report something that identifies the actual code, such as a build hash or a source revision, separately from the platform version number. Without that, the fleet's reported versions become fiction during exactly the period when you most need them to be facts.

Record it in the release log at the time. Not afterwards. The mapping from the new number to the original build is the only thing that keeps this comprehensible, and it has to be written while people still remember.

The bookkeeping hazard this creates

The technique works. What it breaks is the assumption that a version number tells you what code is running, and that assumption is embedded in more places than anyone expects.

Two different numbers now mean the same code. Any analysis that groups by version, defect rates, crash clustering, battery performance, telemetry comparisons, will treat one population as two.

The ordering becomes misleading. A higher number no longer means newer code. The relabelled image is numerically ahead of the bad build and chronologically behind it. Anyone reasoning about "later than" from the number alone draws a wrong conclusion.

The real fix has to clear the relabelled number. When the actual fix arrives it must be numbered above the relabel, not above the original bad build. That is easy to get wrong, and getting it wrong means the fix is silently rejected by the same rule you worked around.

Anything keyed to version breaks subtly. Feature flags, server-side compatibility checks, migration logic, support scripts, documentation. Every one of those that branches on a version comparison now has a case it was never written for.

A partial rollout leaves permanent ambiguity. Some devices got the bad build, some got relabelled good code, some never moved. Months later the version numbers alone cannot distinguish those groups, which is why the separate build identity reported by the device matters so much.

The mitigation is not cleverness, it is explicitness. Treat the relabel as an entry in the release record with its own identifier, its own note explaining what it contains, and its own line in whatever table people consult. Make the device report build identity independently of version. And plan the number sequence deliberately rather than reaching for the next available integer under pressure.

Draw the recovery boundary honestly

Forward-versioned rollback restores an important capability, and it is tempting to then describe the fleet as recoverable. That claim needs a boundary drawn around it, written down before you need it.

An update can fix anything on a device that still checks in. It boots, it connects, it fetches, it installs. For this population an update is a real remedy and the technique above is what makes it available on a monotonic platform.

An update cannot fix a device that does not reach the update channel. If the bad build does not boot, or boots but cannot bring up networking, or crashes before the update agent runs, then no published image reaches it. Not a higher-numbered one, not any. The device is not slow to recover, it is outside the reach of software entirely.

Between those two sits the population that depends on the bootloader. A device that fails to boot the new image and falls back to the previous slot is recoverable, and it is recoverable because of a mechanism that was designed in before shipping rather than because of anything you can publish now. Whether a given unit lands in this group or the previous one was decided at design time.

Below that line is a bench. Physical access, a programming interface, a person and a return path. For a consumer fleet that is effectively a write-off. For a small industrial deployment it is a cost you can quantify.

Writing this down in advance changes the conversation during an incident, because the question "can we fix this remotely" has a prepared answer that depends on which stage the failure occurred in rather than on optimism. It also changes design decisions beforehand, because once the boundary is explicit, the value of a bootloader that can fall back without network access becomes obvious to people who were previously treating it as an optional refinement.

The honest limit

This technique is a recovery mechanism, not a release strategy. Every use of it is a small permanent cost to the clarity of your version history, so the correct frequency is rare. A team that reaches for it routinely has a testing problem rather than a versioning problem.

It also does nothing about state. If the bad build migrated stored data to a format the older code cannot read, then installing the older code gives you a device running proven firmware against data it does not understand, and the symptoms can be worse than the original defect. Forward-versioned rollback restores a binary. It does not restore a schema. Before using it, somebody has to answer whether the bad build changed anything persistent, and if it did, the relabelled image may need to be able to read the newer format, which means it is no longer the untouched binary and the whole argument weakens.

And it depends on the platform comparing numbers rather than tracking artefact lineage. A platform that recorded which build each device had previously run could reject the relabel on other grounds. The technique is a property of how a particular rule is implemented, not a general guarantee, and it is worth confirming it still works before you need it rather than during an incident.

FAQ

Why do some platforms refuse firmware downgrades? Because accepting a lower version is a rollback attack. A party who can influence what a device fetches could serve a correctly signed older image with a vulnerability that has since been patched. The rule also makes state migration one-directional, since firmware rarely implements reverse migrations, and it lets the platform make simple true statements about minimum fleet version.

How do you roll back on a platform that enforces a strictly higher version? Forward-versioned rollback. Take the known-good older binary, change only its version metadata so the number sits above the bad build, sign it and publish it as a normal update. The platform sees an upgrade and installs it, and the device runs the old proven code under a new number. Do not rebuild from source, because the binary's field history is the whole point.

What does this break? The correspondence between version numbers and code. Two numbers now mean the same code, a higher number no longer means newer code, the real fix must be numbered above the relabel rather than above the bad build, and anything that branches on a version comparison acquires an untested case. The defence is a separately reported build identity and a release log entry written at the time.

Should the relabelled image use the next consecutive version number? No. Leave a visible gap above the bad build, so the real fix has room and the anomaly is obvious to anyone reading the version list later. Taking the next integer under incident pressure is how a relabel becomes indistinguishable from an ordinary release six months afterwards.

What can a firmware update actually recover? Any device that still boots, connects and reaches the update channel. It cannot recover a device whose current build fails to boot, fails to bring up networking, or crashes before the update agent runs, because no published image gets to it. Between those sits the population saved by a bootloader that falls back to the previous slot, which is a design decision made before shipping.

When should this technique not be used? When the bad build changed persistent state. Installing older code against migrated data gives you proven firmware that cannot read its own storage, and the symptoms can be worse than the original defect. It is also the wrong answer if it is being used often, since frequent rollbacks indicate a gap in the pre-release gates rather than a versioning problem.