The soak period is derived rather than chosen: it has to outlast the slowest failure you are trying to catch, and give every unit at least two chances to report, because one missed check-in is noise and two consecutive is a state.
| Stage | Units | Cumulative | Share of fleet | Halt below | Elapsed |
|---|---|---|---|---|---|
| Stage 1 | 50 | 50 | 0.1% | 89.8% | 24 h |
| Stage 2 | 250 | 300 | 0.8% | 93.8% | 2.0 days |
| Stage 3 | 1,250 | 1,550 | 3.9% | 95.6% | 3.0 days |
| Stage 4 | 6,250 | 7,800 | 19.5% | 96.4% | 4.0 days |
| Stage 5 | 31,250 | 39,050 | 97.6% | 96.7% | 5.0 days |
| Stage 6 | 950 | 40,000 | 100.0% | 95.3% | 6.0 days |
The smallest check-in shortfall a cohort can resolve
Three standard deviations below baseline, by cohort size and healthy check-in rate. Read it as: a failure affecting fewer units than this will not move the cohort's check-in rate far enough to be distinguishable from an ordinary quiet day. This is why a very small first cohort is not the cautious choice people take it for.
| Cohort size | 90% baseline | 95% baseline | 97% baseline | 99% baseline |
|---|---|---|---|---|
| 10 | 28.5 pts | 20.7 pts | 16.2 pts | 9.4 pts |
| 25 | 18.0 pts | 13.1 pts | 10.2 pts | 6.0 pts |
| 50 | 12.7 pts | 9.2 pts | 7.2 pts | 4.2 pts |
| 100 | 9.0 pts | 6.5 pts | 5.1 pts | 3.0 pts |
| 500 | 4.0 pts | 2.9 pts | 2.3 pts | 1.3 pts |
| 1,000 | 2.8 pts | 2.1 pts | 1.6 pts | 0.9 pts |
| 5,000 | 1.3 pts | 0.9 pts | 0.7 pts | 0.4 pts |
Red is over 15 points, which means only a catastrophic failure is visible. Amber is over 6. Resolution improves with the square root of cohort size, so ten times the units buys about three times the sensitivity.
A worked example
A 40,000 unit fleet, checking in every six hours, with a healthy rate of 97 percent because units are routinely powered down or out of coverage. The team plans a canary of 50 units, then 250, then 1,250, and so on, with an hour between stages so the rollout finishes inside a working day.
Two things are wrong and neither is visible from the plan. The soak is one hour against a check-in interval of six, so most of the canary has not reported even once when the next stage begins. And at 50 units against a 97 percent baseline, the cohort can only resolve a shortfall of about 7 percentage points, so a defect bricking one unit in twenty-five passes the canary unnoticed.
Deriving the soak instead of choosing it gives 24 hours here, because the failure class they are worried about needs a full duty cycle to appear and that dominates the two check-in intervals. Six stages at 24 hours is six days rather than a working day, which is the real cost of the protection and worth stating before somebody shortens it under release pressure.
The number that settles the argument is the tolerance. Half a percent of 40,000 is 200 units, and the cumulative schedule runs 50, 300, 1,550, 7,800, and so on. So exactly one stage of six stays inside what they said they could afford to lose. From stage two onward the rollout is committing more of the fleet than the tolerance allows, which means the remaining stages are pacing rather than protection.
That is a shape problem, not a soak problem. Dropping the growth factor from 5x to 2x buys one more stage inside the tolerance and costs four extra days. Halving the first cohort as well, to 25 units at 2x, buys three stages inside the tolerance for eleven days total. Whether either trade is worth making is a real decision, and it is one nobody can have while the schedule is a list of cohort sizes with no tolerance written next to it.
The arithmetic, so you can check it
Cohorts grow geometrically until they cover the fleet. The soak period is max(failure manifest time, 2 x check-in interval), because a stage cannot conclude anything before every unit has had more than one chance to report, and one missed check-in is noise while two consecutive is a state.
The halt threshold is a one-sided normal approximation to the binomial: baseline - 3 x sqrt(baseline x (1 - baseline) / n). The same expression, read the other way, gives the smallest shortfall a cohort can resolve, which is the reference table above.
This assumes check-ins are independent, and they are not. A regional outage, a carrier problem or a weekend takes out a correlated block of units and looks exactly like a bad image. That is an argument for cohorts spread across regions and carriers rather than for a looser threshold, and it means a halt should be investigated rather than automatically rolled back. The threshold tells you when to look, not what you found.
The failure modes this is protecting against are in the update that bricks the fleet, and why absence is the only honest fleet signal in ship it once, reach it never.
Questions
By cohort, growing geometrically, with a soak period between stages long enough for the relevant failure class to appear and a halt condition defined as a number before the rollout starts. The first cohort should be small enough that losing all of it is survivable, and large enough to resolve the failure rate you care about, which are competing requirements worth resolving deliberately.
Check-in rate against expected, not error rate. A bricked device sends nothing at all, so a system watching for errors sees a clean rollout while units go dark. Track time since last successful contact per unit, expect a check-in on a known interval, and treat sustained absence as the signal.
Longer than the slowest failure class you are trying to catch, and at least two check-in intervals so every unit has had more than one chance to report. If a configuration migration only fails on units that have been through a full duty cycle, an hour between stages provides no protection at all regardless of how many stages there are.
From the statistics of the cohort, not by intuition. A healthy fleet never checks in at 100 percent, so the question is what shortfall is large enough to be a signal rather than the usual population of units that are offline today. Three standard deviations below your baseline rate, computed for that cohort size, is rare enough not to fire on a quiet weekend.
Because statistical resolution scales with the square root of the cohort size. Fifty units can only resolve a check-in shortfall of roughly seven percentage points, so a defect bricking one unit in fifty passes it unnoticed. The first stage feels like the safest one and is frequently proving much less than it appears to.