The Update That Bricks the Fleet: OTA Failure Modes

Mostafa Dhouib·September 7, 2026·7 min read

The Update That Bricks the Fleet: OTA Failure Modes

The short answer. Over-the-air update is the one operation where a defect can destroy the mechanism that would fix the defect. Everything about the design follows from that: the updater must be simpler than what it updates, rollback must not depend on the new image working, the device must recover from power loss at any instant, and the rollout must be staged so a bad image reaches a handful of units rather than all of them. Getting this right is unglamorous and it is the difference between a bad week and a fleet you cannot reach.

Most software failures are recoverable because you can deploy again. Firmware update is the exception: it is the deploy mechanism, and a failure in it can leave a device that no longer accepts deploys.

That asymmetry justifies a level of care that looks disproportionate right up until the moment it is not.

The failure modes, in order of how much they cost

Power loss mid-write. The most common and the most predictable. A device losing power partway through a flash write has a partially written image. If that region is what boots, the device is dead.

A valid image that does not work. Signature checks pass, the image is complete, and the device boots into firmware with a bug that prevents it from connecting, or from accepting the next update. Every integrity check in the world passes and the outcome is the same as a corrupt image.

A rollout that reaches everything at once. The bad image is on the whole fleet before anyone has evidence it is bad. This converts a defect into an event.

Configuration and data migration. The new firmware expects a different configuration format and the migration fails on units in an unexpected prior state. Usually the oldest units, which are also usually the ones you least want to lose.

A downgrade that is blocked when you need it. Version rules that prevent downgrade, added for good security reasons, and now the rollback you need is refused by your own policy.

The four requirements that follow

One: the updater must be simpler than what it updates

The component that performs the update is the one that must never fail. So it should be small, rarely changed, and doing as little as possible: verify a signature, write to the inactive slot, set a flag, reboot.

Every feature added to the bootloader is a feature that can brick the fleet. That is a very unusual risk profile and it should make the bootloader boring on purpose. Where it must be updated, it deserves a separate, more conservative process than application firmware.

Two: two slots, and rollback must not depend on the new image

An A/B scheme with a confirmation step is the standard answer and each part carries weight:

Write to the inactive slot. The running image is untouched throughout, so power loss during the write leaves the device booting the old firmware.

Boot the new slot provisionally. Marked as trial rather than as current.

The new firmware must confirm itself, by doing something that proves it works: connecting to the backend, passing a self-test, accepting a command. Not merely by starting.

A watchdog reverts if confirmation does not arrive. This is the property that handles the valid-but-broken image, and it is the one most often left out. Without it you have integrity checking, which was never the hard problem.

The confirmation criterion is where judgement lives. Confirming on boot proves almost nothing. Confirming on a successful backend connection proves the thing you actually need, which is that you can still reach the device.

Three: recover from power loss at any instant

The test is not that it usually works. It is that there is no instant at which power loss leaves an unbootable device.

That means writes ordered so the flag that selects the boot slot is set last and atomically, and it means testing by actually cutting power, repeatedly, at random points through the update, on real hardware. Automating that with a switchable supply and a few hundred cycles finds things no code review does.

Four: stage the rollout and define what stops it

A defect reaching ten units is a bad day. Reaching a hundred thousand is a different category of event, and the only thing separating them is rollout policy.

Stage by cohort, with a soak period between stages long enough for the failure to appear. If a class of failure takes a day to manifest, an hour between stages provides no protection.

Define the halt condition before the rollout, as a number: check-in rate for the cohort, error rate, confirmation-failure count. Written down, with a name against it, so the decision is not made under pressure by whoever is awake.

Keep the previous image available and make sure the rollback path does not depend on anything the new image changed.

The signal that matters most is absence

The unit that fails to come back after an update is the one you need to know about, and it cannot tell you anything, because it is the one that is broken.

So the metric is not error rate. It is check-in rate against expected: how many units in the cohort reported after the update versus how many should have. A device that bricked sends nothing, and a system watching only for errors sees a clean rollout.

This is the same structural blindness as a fleet platform that cannot detect a silent node, and it has the same fix: track time since last contact per unit, expect a check-in, and treat its absence as the signal.

The property that should shape the whole product

An update mechanism is not a feature of connected hardware. It is the thing that determines whether every other decision is reversible.

A device with a reliable update path can ship with known imperfections, because they can be fixed. A device without one has to be right at manufacture, which is a far more expensive standard and one that no amount of testing fully reaches.

Which means the update mechanism should be built and hardened first, before the features that will depend on it, and it should be the most tested code in the product. That ordering feels wrong to everyone and it is correct, for the same reason that designing for hardware you cannot reach puts the recovery path before the capability.

FAQ

What makes OTA firmware update different from software deployment? It is the only operation where a defect can remove your ability to fix the defect. A failed deploy in most software is a bad afternoon; a failed firmware update can leave a device that no longer accepts updates, and the recovery cost is a truck roll per unit or a total loss.

How does A/B slot updating work? The new image is written to the inactive slot while the running image is untouched, then booted provisionally. The new firmware must confirm itself by proving it works, typically by reaching the backend, and a watchdog reverts to the previous slot if confirmation does not arrive.

Why isn't signature verification enough? Because it catches corrupt images, not valid images that do not work. Firmware can pass every integrity check, boot, and still fail to connect or to accept the next update. The confirmation step and watchdog revert are what handle that case.

How should an OTA rollout be staged? By cohort, with a soak period long enough for the relevant failure class to appear, and a halt condition defined as a number before the rollout starts: check-in rate, error rate, or confirmation failures, with one person named to call it.

What should be monitored during a fleet update? Check-in rate against expected, not error rate. A bricked device sends nothing at all, so a system watching only for errors sees a clean rollout while units go dark. Absence is the signal.