A Staged Rollout Of A Bad Build Is Still A Bad Build

Mostafa Dhouib·10 min read

A Staged Rollout Of A Bad Build Is Still A Bad Build

The short answer. A rollout ladder, an allowlist of internal devices, then one percent of the fleet, then wider, controls how many devices a build reaches. That is the whole of what it does. It does not inspect the build, does not test it, and cannot make it safe. A one percent rollout of firmware that bricks when power is lost mid-install bricks one percent of the fleet, and those devices do not come back when you halt the rollout. The ladder limits the blast radius of a reversible failure and does nothing at all about an irreversible one. The gate that asks whether this build survives an interrupted update has to run before device number one, and an absent answer is a fail, not a reason to proceed carefully.

Staged rollout is the correct practice and I recommend it on every connected-hardware engagement. The problem is what people believe it is doing for them.

It is a containment mechanism. It is routinely treated as a verification mechanism, and those are not the same thing.

What the ladder actually controls

Strip the ceremony and a rollout ladder is one variable: the size of the set of devices eligible to receive this image right now.

The allowlist stage restricts eligibility to devices you own and can physically reach. If something goes badly wrong here, the remedy is a cable and a person in the same building.

The small percentage stage opens eligibility to a sample of the real fleet. These are devices in real conditions with real power supplies and real network quality, which is genuinely information you cannot get on a bench.

The wider stages increase eligibility progressively while you watch the telemetry.

Every stage answers one question, which is how many. None of them answers what happens, because the ladder has no opinion about the contents of the image. Promotion between stages is gated by observing the previous stage, which means the ladder is a mechanism for learning from damage rather than for preventing it. That is a reasonable thing to be, as long as the damage is the kind you can undo.

The property that decides whether a ladder helps

The question that determines whether staged rollout is protection or theatre is whether the failure is reversible on the affected devices.

Reversible failures are exactly what a ladder is for. A feature regression, a crash loop that still permits the update agent to run, higher battery consumption, a connectivity bug that does not stop the device checking in. In every one of these the device is still reachable, so limiting exposure limits how many users were inconvenienced for how long, and the fix arrives through the same channel. A small first stage genuinely converts a fleet-wide incident into a contained one.

Irreversible failures make the ladder arithmetic rather than protection. An image that fails to boot and cannot fall back. A partition table written incorrectly. An install path that bricks the device when power is interrupted. Here the ladder does not reduce harm, it only chooses the coefficient. One percent of the fleet is still a permanent number of dead units, and halting the rollout at the first report does nothing for any device that has already received the image.

The uncomfortable consequence is that the ladder gives the most reassurance in exactly the case where it provides the least benefit. A small first stage feels cautious. Against a bricking defect it is a decision about how many devices to destroy, taken by people who believe they are being careful.

The gates that have to pass before device one

If the ladder cannot verify the build, something else has to, and it has to happen before any device outside your own hands receives the image. These are the gates whose absence should stop a release.

Power loss at an arbitrary point during install. Not a graceful shutdown. Supply removed abruptly, repeatedly, at randomised points through the write, and the device must come back on either the old image or the new one every single time. This is the gate that is most often skipped, because it needs a fixture that cuts power on command, and it is the one that corresponds directly to the irreversible failure mode.

Network loss at an arbitrary point during download. The device must resume or abandon cleanly, with no partial image treated as valid, and must still be running the old code afterwards.

Rollback without the new image cooperating. The fallback path must be exercised by actually booting a deliberately broken image, not by reading the bootloader source and agreeing that it looks correct. A fallback that has never executed is a fallback with unknown status.

The upgrade path from every version in the field. Not just from the latest. If the fleet contains several older versions, each one is a distinct starting state with its own migration, and at least one of them is usually the one nobody tested.

Persistent state migration, forward and under interruption. Including what happens if the device is interrupted midway through a migration, which leaves storage in a state neither version expected.

The lowest supply condition you will ship into. A cold cell near end of life has higher internal resistance, so an install that works on a bench supply can sag below the brownout threshold in the field. The install is the highest sustained current the device will ever draw, which makes it the worst case for exactly this.

Every one of those is a question with a yes or no answer, obtainable on a bench, by a person, before anything is published. None of them requires a single field device.

Missing evidence is a fail

This is the part that needs stating bluntly, because it is where the process usually gives way.

The question at a release gate is not "do we have a reason to think this is broken". It is "do we have evidence that it survives an interrupted install". Those are different questions and they default in opposite directions. The first defaults to proceed, because the absence of a known defect is the normal state before a defect is found. The second defaults to stop, because absence of evidence is the normal state before anybody does the work.

The arguments for treating a gap as a pass are always the same ones, and they are always wrong in the same way.

"We have not changed the update path." Then the evidence from the last time it was tested still applies only if nothing in the image affects write timing, power draw, partition layout or startup sequence. That is a claim about the diff that somebody should have to make explicitly.

"We will catch it in the one percent stage." You will catch it. You will not fix it, because the devices that found it are bricked, and the information arrives after the harm.

"The platform handles install integrity." Partly, and the part it handles is documented. What the platform does not handle is your migration, your startup sequence, and your own power behaviour during the write.

"We are already late." This is the real reason most of the time, and it deserves a direct answer: an interrupted-install failure costs more than the delay, it costs it irrecoverably, and it costs it in units rather than in days.

So the gate has to be a list with an owner and a date against each line, and a blank line has to block the publish. Not a conversation about likelihood. If nobody has pulled the power during a write on this build, that line is a fail.

Where the ladder earns its place

None of this is an argument against staged rollout, and it would be a poor conclusion to draw.

Once the irreversible failure modes have been gated out on a bench, the remaining risk is mostly the risk you cannot reproduce: the distribution of real network conditions, real cell states, real usage patterns, real device revisions, real ambient temperatures. A small first stage samples that distribution at a cost you can absorb, and that is information no bench produces.

The ladder is also the only mechanism that gives you time. Promotion gated on observed stage-level telemetry, with an explicit halt criterion written before the rollout starts, means a slow-burning problem gets found while the affected population is small. The right mental model is that the bench gates handle what you can enumerate, and the ladder handles what you cannot, and the second only works because the first already removed everything irreversible.

Order matters more than either mechanism. A ladder running ahead of the gates is a sampling procedure for discovering bricks.

The honest limit

Bench gates are only as good as the list, and the list is written from what the team can imagine. A failure mode nobody thought of does not appear on it, and the first population to meet it will be a field population regardless of how disciplined the release process is. That is the genuine case for a staged rollout and it is also the reason the gates can never be declared complete.

Power-cut testing in particular reproduces a condition, not a distribution. Pulling a bench supply is not identical to a cell sagging under load at low temperature, and a fixture that cuts cleanly may miss failures that happen in the slow brownout region where the processor is partially functional. Testing the slow sag is harder and it is closer to what actually happens.

And there is no model in any of this. It is a list of enumerated conditions, a fixture, and a count of pass or fail. That is precisely why the missing-evidence rule is enforceable, because every line has an answer somebody either obtained or did not.

FAQ

Does a staged rollout make a firmware release safe? No. It controls how many devices a build reaches and nothing else. It does not inspect or test the image. Promotion between stages is gated on observing the previous stage, which makes it a mechanism for learning from damage rather than preventing it. That is useful when the damage is reversible and meaningless when it is not.

What happens if you roll out a bricking build to one percent of a fleet? One percent of the fleet bricks, permanently. Halting the rollout stops further devices receiving the image and does nothing for the devices that already have it. In that situation the stage percentage is not a safety margin, it is a decision about how many units to destroy.

Which failures does a staged rollout actually protect against? Reversible ones, where the device remains reachable through the update channel. Feature regressions, battery regressions, crash loops that still allow the update agent to run, connectivity bugs that do not stop check-in. For these, a small first stage converts a fleet-wide incident into a contained one and the fix arrives the same way the defect did.

What must be tested before any field device receives a build? Abrupt power loss at randomised points during install, network loss during download, rollback exercised by actually booting a deliberately broken image, the upgrade path from every version still in the field, persistent state migration including interruption partway through, and install behaviour at the lowest supply condition you will ship into.

Why should missing test evidence block a release? Because the question is whether there is evidence the build survives an interrupted install, not whether there is a known reason to think it is broken. Those two questions default in opposite directions. Absence of a defect report is the normal state before a defect is found, and absence of evidence is the normal state before anyone does the work.

Is there any point in a rollout ladder once the bench gates pass? Yes, and that is when it becomes valuable rather than decorative. What remains after the enumerable failures are gated out is the distribution you cannot reproduce: real network quality, real cell states, real device revisions, real temperatures. A small first stage samples that at an absorbable cost, and it buys time to halt on a slow-burning problem.