Ninety Percent Is Not Success: Update Telemetry That Measures The Wrong Stage

Mostafa Dhouib·10 min read

Ninety Percent Is Not Success: Update Telemetry That Measures The Wrong Stage

The short answer. On at least one major connected-device platform, an update that reports 90 percent progress has finished downloading the image. It has not installed it, not rebooted into it, and not confirmed it is running. Teams read 90 percent as nearly done, which is understandable and exactly backwards, because the download is the safe part and everything dangerous happens after that figure stops moving. A device sitting at 90 percent is not a device about to succeed, it is a device at the start of the risky stage. The fix is not a better progress bar. It is to stop reporting one number and report four independent stages: download complete, install complete, reboot into the new image, and a version confirmation from the device itself.

A rollout dashboard showed a long tail of devices at 90 percent. The reading in the room was that they were nearly there and would catch up.

They were not nearly there. They had done the easy part and stopped.

Where the figure comes from

The progress number is reported by the platform's update agent, and it describes the part of the process the agent can see with confidence, which is the transfer. Bytes received over bytes expected is a genuine measurement with a known denominator, so it maps cleanly onto a percentage.

What follows the transfer does not map cleanly onto anything. Verifying a signature either succeeds or fails. Writing an image to the inactive flash slot has a duration but no meaningful intermediate state. A reboot is a discontinuity, not a progression. Confirming the new version is running is a single binary fact.

So the platform does the only honest thing available to it. It scales the transfer across most of the bar, because the transfer is the only part with a continuous measure, and it leaves a small remainder for everything else. The number is not wrong. It is answering a question about bytes, and it is being read as an answer to a question about outcomes.

Why this is worse than no telemetry

If a dashboard showed nothing after the transfer, every team would know they had a blind spot, and somebody would build a check. Instead the dashboard shows a high number, and a high number is read as progress toward success.

It converts absence of information into apparent good news. The reason a device sits at 90 percent is that nothing has been heard from it since the download finished. That is the signature of a device that failed during install, failed to boot, or is bricked. It is displayed as almost finished.

It fails most reassuringly in exactly the worst case. A device that cannot complete the install is a device that has stopped reporting, and a device that has stopped reporting holds its last value. The population you most need to see is the population rendered least visible.

It sets the wrong alert thresholds. If 90 percent means nearly done, a dashboard mostly at 90 percent looks fine, so nobody alerts on it. The condition that should trigger an immediate halt looks like the condition that should trigger patience.

It corrupts the postmortem. After an incident, the record says devices reached 90 percent, and that sentence will be read for years as evidence that the update nearly worked.

The general form is that telemetry which measures the wrong stage is more dangerous than no telemetry at all, because the cost of a blind spot is a known unknown, and the cost of a misaimed metric is a false belief that people act on. No telemetry makes you cautious. Wrong telemetry makes you confident.

The four stages, reported separately

The replacement is not a more granular percentage. It is to recognise that an update is a sequence of distinct transitions, each with its own failure mode, and to report each one as its own fact.

Stage one: download complete and verified. The image arrived and its signature and checksum validated against what the server intended to send. Failures here are network failures, truncated transfers and corrupted images, and they are the cheapest class, because nothing has been written yet and the device is still running known-good code. This is where almost all of the progress bar lives and almost none of the risk.

Stage two: install complete. The image has been written to the inactive slot and the slot has been verified after writing. Failures here are flash write errors, insufficient space, and power loss during the write. This is the stage where a device can be damaged, and on most platforms it is the stage nothing reports.

Stage three: rebooted into the new image. The device restarted and the bootloader selected and started the new slot. Failures here are the expensive ones: an image that does not boot, a bootloader that cannot fall back, a configuration migration that fails before networking comes up. A device that fails here cannot tell you that it failed here.

Stage four: version confirmed by the device. The device, running the new code, independently reports its own version string, and that string matches the target. This is the only stage that constitutes success, because it is the only one reported by the thing you were trying to change, after the change, using the code you shipped.

Each stage needs its own count in the dashboard, its own timeout, and its own alert. A device that completes stage one and does not complete stage two within a defined window is not in progress. It is a failure awaiting classification.

What the decomposed view shows you that the bar cannot

Four counters produce diagnostic shape that a single number destroys.

Which stage the failures cluster in names the cause. Mass failures at stage one point at the network, the server or the image size. At stage two, at flash handling or power. At stage three, at the image or the bootloader. At stage four, at reporting or at a device that boots but cannot connect.

The gaps between stages are the signal, and they are absences. Devices that entered stage two and never reported stage three are the population that matters most, and the only way to see them is to subtract one counter from another and alert on the difference. Success counts never reveal this. Only the arithmetic between stages does.

Stage four closes the loop with independent evidence. A server believing it sent version two is not the same as a device saying it is running version two. Only the second is a fact about the fleet, and insisting on it is what turns a rollout from a hope into a measurement.

Per-stage timing catches degradation early. If the time from stage two to stage three starts drifting upward across a rollout, something is wrong before anything has failed outright.

Making the device cooperate

Decomposed telemetry requires the device to emit the events, and some of them have to survive conditions in which the device is barely functional.

The stage-two report has to be written before the reboot, because after the reboot the device may not be in a position to tell you anything. The stage-four report has to be the first meaningful thing the new image does once it has a connection, before any feature initialisation that could fail and prevent it. And the device needs a persistent record of where it was in the sequence, so that a unit which fell back to the old image can report that it attempted and rejected a version rather than appearing to have never started.

The pattern is that every transition which can fail silently needs an explicit positive report on the far side of it, and the report must depend on as little functioning code as possible.

The honest limit

Decomposition improves what you can see. It does not reduce what can go wrong, and the stage that matters most remains the hardest to instrument, because a device that will not boot cannot send you anything. Stage three is always inferred from the absence of stage four, and inference from absence requires a timeout, which means a judgement about how long is too long. Set it short and you declare healthy devices failed. Set it long and you discover a bad rollout late.

There is also a floor set by the platform. If the platform's own agent owns the install and reboot, your device-side code may genuinely not get an opportunity to report between those steps, and in that case the decomposition you can build is coarser than the one described here. You work with what the platform exposes and you make the gap explicit rather than pretending it is covered.

And none of this is a safety mechanism. Knowing precisely which stage a device died in does not revive it. Good telemetry shortens the time between a bad build and the decision to stop, which is valuable and is not the same thing as a build that does not brick devices. The gate that decides whether an image survives an interrupted install runs before the rollout starts, and no amount of staged monitoring substitutes for it.

FAQ

What does 90 percent mean on a firmware update progress bar? On at least one major connected-device platform it means the download completed, not that the install completed. The transfer is the only part of the process with a continuous measure, so it occupies most of the bar, and everything after it is compressed into the remainder. A device at 90 percent has finished the safe part and is entering the risky part.

Why is a dashboard full of devices at 90 percent a bad sign? Because the reason a device holds that value is that nothing has been heard from it since the download finished, which is the signature of a failed install or a failed boot. The metric holds its last reported value, so the devices in the most trouble look the closest to done.

How should firmware update progress be reported instead? As four separate stages with their own counters, timeouts and alerts: download complete and verified, install written and verified, rebooted into the new image, and version confirmed by the device itself. Only the fourth constitutes success, because it is the only one reported by the device after the change using the new code.

Why is misaimed telemetry worse than no telemetry? Because a blind spot makes people cautious and a wrong metric makes them confident. A missing measurement prompts someone to build a check. A high number that measures the wrong stage gets read as progress toward success, sets alert thresholds in the wrong place, and ends up in the postmortem as evidence the update nearly worked.

How do you detect devices that failed during install or boot? By arithmetic between stages rather than by counting successes. Devices that reported install complete and never reported a confirmed version are the population at risk, and they are visible only as the difference between two counters. That requires a timeout, which is a judgement: too short declares healthy devices failed, too long finds a bad rollout late.

What has to be true on the device for this to work? It must report install completion before the reboot, report its version as the first meaningful action of the new image once it has a connection, and keep a persistent record of where it was in the sequence. That way a unit which rolled back reports an attempted and rejected version rather than appearing never to have started.