A consumer hardware brand ships connected devices across several product lines, built in more than one factory, updated over the air into people's homes. Each update is a moment where a working device can become a paperweight. When that happens there is no technician on site and no way back over the network. There is a return, a refund, a support ticket, and a review that outlives the fix.
An update is five steps, and only one of them is the download.
From the outside an over-the-air update looks like a single event. A new version appears and the device carries on. Underneath, it is a sequence: download the image, check that it is intact and genuine, write it to storage, restart into it, and report the new version back. Each step can fail independently, and each failure looks different.
The step that decides whether a failure is survivable is the write. Some modules keep two copies of the software, one running and one being updated, and only switch over once the new copy is verified. If power fails during the write, the device restarts into the old copy as though nothing happened. Others keep a single copy and overwrite it in place. If power fails during that write, there is no complete copy anywhere, and the device cannot start.
Whether a module has one copy or two is a property of the silicon and how it was configured, not of the update service. That is why the same update process can be perfectly safe on one product and destructive on another, and why an update strategy has to be judged against the specific hardware it is updating.
The brief asked for someone to run the release process: an allowlist, then one percent of the fleet, then wider. That process is real and worth having. It also answers only one question, which is how many devices a build reaches. It says nothing about whether the build survives being interrupted halfway through installing. A careful rollout of a build that bricks on power loss bricks its share of the fleet, neatly and on schedule.
Updates rarely fail where the dashboard is looking.
Most update tooling is built to answer one question well: did the device receive the file. That is the easiest thing to measure and the least likely to go wrong. The failures that cost money happen afterwards, during installation and restart, where the tooling often has nothing to say.
The second pattern is a false sense of safety from the rollout itself. A staged rollout is good engineering and it reduces how many devices a bad build can harm. Teams then reason that because the rollout is careful, the release is safe. Those are different claims, and the gap between them is filled by every device in the first rung.
The reflex, and the fix.
Treat the rollout ladder as the safety mechanism
It is the industry-standard answer. It genuinely limits how far a bad build travels, and it is visible, procedural and easy to report on.
Limiting reach and being safe are different properties. A ladder can only tell you a build is bad after it has already damaged the devices it reached.
Prove the build survives interruption, before device one
A gate that runs before the ladder and blocks it, built from a failure assessment specific to the module's silicon and an executable set of injected failures.
A build that has been shown to recover from a botched update, plus a written line between what an update can fix and what needs a person at a bench.
The ladder assumes the build is safe. So the work starts one layer down, by checking that assumption before anything relies on it.
A survivability gate that runs before the ladder
Before any device sees the build, the module's update path is assessed for how it behaves when power disappears at each stage. The assessment is specific to the part family, because recovery behaviour depends on the exact silicon. It ends in an executable go or no-go, and the ladder cannot start without a go.
Reading the progress bar correctly
On this platform a high progress figure means the download finished. Installing, rebooting and reporting the new version all happen afterwards, and that is where devices fail. So a unit stuck near the end of the bar is diagnosed at those later stages first.
Binding each image to the right product
Several product lines share one update platform. Images are bound to a product identity so the wrong build cannot reach the wrong device, which is the failure that turns a multi-product fleet into a multi-product incident.
A ladder that advances on evidence
The rollout moves up a rung only when field data shows the current rung is healthy. A date on a calendar never advances it, and an absence of data counts as a failure. Unknown is never promoted to safe.
Rolling back on a platform that forbids going backwards
The platform only accepts versions higher than the one installed. So rollback means taking the last known-good binary, giving it a higher version number, and shipping it as an ordinary upgrade. The device runs the old, safe code because it believes it is moving forward.
What each choice cost, and what it bought.
APutting the gate below the ladder
The obvious place for safety work is the rollout, because that is where releases are visibly managed. Putting a gate underneath it changes what the rollout is for. The ladder stops being the thing that discovers bad builds and becomes a way of catching what the gate could not foresee, mainly the gap between a bench and real homes. The gate handles what can be known in advance. The ladder handles what can only be learned in the field.
BJudging the update path per part family
It would have been simpler to assess the update service once and treat every product the same. Recovery behaviour is set by the silicon, though, and different product lines used different module families. Assessing each family separately is more work, and it is the only version where the answer actually applies to the device in the box.
CTreating missing data as a failure
A rollout that advances when nothing bad is reported will advance through a silent problem, because devices that have stopped working also stop reporting. Requiring positive evidence of health before each step means a quiet rung holds the release. That is slower in the good case and it is the only safe behaviour in the bad one.
DDrawing the recovery boundary in writing
Some failures can be fixed by sending another update. Some cannot. Writing down exactly which is which, per product, means nobody promises a customer an over-the-air fix that does not exist, and the support team knows from the start which devices need to come back.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
The release held at 2 percent is a measured, delivered outcome.
A textbook staged rollout, executed perfectly, that still bricks its share of the fleet at every rung, because nothing upstream of the ladder ever asked whether the build survives an interruption.
A gate before the first device, built on the real recovery behaviour of the real silicon. A ladder that only climbs on evidence. A rollback that works on a platform built to prevent rollbacks. And a release that was held at 2 percent the moment field data disagreed with the bench.
What it owns, and what it hands to a person.
Some failures cannot be recovered over the air, and the recovery boundary says so plainly. A single-bank module that loses power at the wrong moment needs a bench, and no update strategy changes that. The release held at 2 percent is the other half of the honest picture. Field data showed updated units dropping offline more often than the baseline, which the bench had never shown. The bench is necessary. It is not sufficient.
Any procedure that limits damage depends on an assumption about the thing it is limiting, and that assumption is usually where the risk lives. Ask what the procedure takes for granted, then go and prove it one layer down. Staged deployments, canary releases, feature flags and phased migrations all share this shape.
A staged rollout limits how many devices a bad build reaches. It does not make the build safe, and treating it as a safety mechanism hides the risk it cannot see.
Whether a failed update is survivable is decided by the hardware, specifically whether the module keeps one copy of its software or two. Judge the update path against the actual silicon.
Progress bars usually measure the download, which is the step least likely to fail. Monitor installation, restart and version reporting separately.
A platform that only accepts higher versions can still roll back, by relabelling a known-good build with a higher number. Work this out before the incident, not during it.
Bench testing is necessary and not sufficient. Design the rollout so field evidence can override it, because the field is where the bench is wrong.