A Queue That Fails Safe

Mostafa Dhouib·9 min read

A Queue That Fails Safe

The short answer. When software controls a physical queue of people, a failure does not produce a data problem. It produces a room full of people who do not know whose turn it is, and no amount of later correction undoes that. So the design priority is not any feature of the central system. It is that each counter terminal keeps issuing and calling numbers from a local ledger when the network is gone, and reconciles when connectivity returns. The failure that catches teams out is the recovery surge: everything that queued up during the outage releasing at once, which produces a second, worse incident a few minutes after the first appears to be fixed.

Queue management reads like a straightforward piece of software. Issue a ticket, maintain an ordering, call the next number, show it on a display.

The thing that makes it different from other software of similar complexity is that its state is also held in the physical world, by people standing in a room, and those people do not roll back.

A failure here is not a data failure

Most systems fail by producing wrong data that is later corrected. This one fails by destroying a shared understanding.

If the display freezes and the ticket printer stops, the queue does not pause. It degrades into whatever social ordering the room can improvise, which at a busy counter means the people who are most assertive go first and the people who were there longest do not. Staff have to arbitrate disputes they have no evidence for, because the evidence was on a screen that is now blank.

Restoring the system does not restore the ordering. The tickets people are holding may no longer match anything the system knows about. Some of those people have already been served. Some have left. The system comes back with a confident view of a queue that no longer exists.

This is why the usual availability conversation is the wrong conversation. A target expressed as a percentage of uptime treats all minutes as equivalent. Here, three minutes of downtime during the quiet period costs nothing and three minutes at peak costs the rest of the day.

Offline-first means the terminal is the system

The correct inversion is that the central service is a convenience and the terminal is the authority for its own counter.

Each terminal holds a local ledger. It issues ticket numbers from a range allocated to it, records every issue and every call locally, and continues to operate when it cannot reach anything. The screen keeps working. The printer keeps printing. Staff keep calling numbers. A network outage becomes invisible to the people in the room, which is the only acceptable outcome.

Two design decisions make this work.

Allocate number ranges, do not request numbers. If a terminal has to ask a central service for the next number, it cannot issue when the service is unreachable. If it holds a pre-allocated block, it issues from the block locally and requests a new block well before the current one is exhausted. The block size is chosen so that a terminal can run through an outage of realistic length without running dry.

Make the local ledger append-only. Every event is appended with a local sequence number, the terminal identifier and a local timestamp. Nothing is updated in place. This is what makes reconciliation possible later, because the terminal can describe exactly what it did in order, and it is also what lets a terminal recover its own state after a power cut without reasoning about partial writes.

Reconciliation on reconnect

When connectivity returns, each terminal has a set of local events the centre has not seen, and the centre may have events the terminal has not seen.

The merge has to be defined in advance, because this is not a situation where a sensible default exists.

The terminal wins on what happened at its own counter. It issued the tickets and it called the numbers. The centre did not observe any of it. Any central state that conflicts is stale by definition, and the centre's role is to accept the terminal's account rather than to adjudicate it.

Clock skew is expected, so order within a terminal, not across terminals. Local timestamps from different devices after an outage are not comparable at the precision that matters. Each terminal's own sequence is reliable. A global interleaving reconstructed from device timestamps is a fiction, and reports built on it will be subtly wrong in ways that are hard to explain later.

Reconciliation is idempotent and resumable. The reconnect itself may fail halfway. Events carry a terminal-scoped identifier so that re-sending a batch cannot duplicate it, and a partially completed sync resumes rather than restarts.

Conflicts are recorded, not resolved silently. Two terminals that both issued the same number because a block allocation was lost is a real conflict with no correct answer. Log it, surface it, and let a human decide. A system that silently picks one has removed the operator's ability to notice that something went wrong.

The recovery surge is the real danger

Here is the failure that is worse than the outage.

During the outage, work accumulates. Terminals hold unsent events. Notification messages queue. Display updates queue. Integration calls to downstream systems queue. Every one of those queues has been filling for the duration.

The moment connectivity returns, all of it attempts to flush simultaneously, from every terminal at once. The central service, which was sized for steady-state traffic, receives the entire backlog in a burst. If it falls over under that load, connectivity is restored and the system is still down, and now the terminals have even more backlog to flush on the next attempt. That is a loop that gets worse rather than better.

The visible consequences are specific and bad. Notifications that should have gone out over an hour all arrive in one minute, so people receive a message telling them it is their turn long after they have been served or have left. Displays flicker through a backlog of states. Downstream systems receive a burst they interpret as a surge of real activity.

Rate-limiting recovery is the mitigation, and it has to be designed rather than added.

Terminals flush at a bounded rate with jitter. Not as fast as possible. A deliberate per-terminal rate, with randomised start offsets so that twenty terminals do not reconnect in lockstep.

Time-sensitive output is dropped, not delivered late. A notification telling someone it is their turn has a validity window. If it has expired, the correct action is to discard it. Delivering an expired instruction is worse than delivering nothing, because the person acts on it.

Backlog has a cap with a defined policy at the cap. Unbounded queues turn a long outage into memory exhaustion. The cap and what gets dropped when it is reached are decisions worth making deliberately, in daylight, rather than by whatever the runtime does when it runs out of room.

The centre applies admission control. It accepts reconciliation traffic at a rate it can sustain and tells terminals to back off, rather than accepting everything and collapsing. The terminals keep working regardless, so a slow reconciliation costs nothing visible in the room.

The honest limit

Offline-first operation protects the queue at each counter. It does not preserve anything that genuinely requires coordination across counters.

If the design routes people between counters based on global load, that routing cannot work during a partition, and the honest answer is that the system degrades to per-counter operation and the routing resumes afterwards. Pretending otherwise produces a feature that fails exactly when it is needed.

Local ledgers also move a durability problem to the edge. A terminal that is lost, stolen or physically destroyed before reconciling takes its unsynced events with it, and no central design recovers them. The mitigation is to keep the unsynced window small during normal operation, which is a different objective from surviving a long outage, and the two have to be balanced rather than solved together.

Finally, none of this addresses the hardest part, which is that a queue of people is held together by their shared belief that the ordering is fair. Software can preserve the record. It cannot preserve the belief once it has been broken, which is the strongest argument for keeping the counter working at all costs rather than for recovering well.

FAQ

Why is a queue management system different from other software of similar complexity? Because its state also exists physically, in people standing in a room, and those people do not roll back. A failure does not produce incorrect data to be corrected later. It destroys the shared understanding of whose turn it is, and restoring the system does not restore the ordering, because tickets people hold may no longer match anything the system knows.

What does offline-first mean at a service counter? That each terminal is the authority for its own counter and the central service is a convenience. The terminal issues ticket numbers from a pre-allocated block rather than requesting each one, records every issue and call in a local append-only ledger, and keeps printing and calling numbers when it cannot reach anything. A network outage becomes invisible to the people in the room.

How should terminals reconcile after an outage? The terminal wins on everything that happened at its own counter, because the centre never observed it. Ordering is done within each terminal's own sequence rather than across terminals, since device clocks after an outage are not comparable at the precision that matters. Reconciliation must be idempotent and resumable, and genuine conflicts should be logged for a human rather than resolved silently.

What is a recovery surge? Everything that queued during an outage releasing at once when connectivity returns. Unsent terminal events, notifications, display updates and downstream calls all flush simultaneously from every terminal, hitting a central service sized for steady-state traffic. If it collapses under that burst, the network is back and the system is still down, with more backlog waiting for the next attempt.

How do you rate-limit recovery? Flush from each terminal at a bounded rate with randomised start offsets so devices do not reconnect in lockstep, cap the backlog with an explicit policy for what is dropped at the cap, and apply admission control at the centre so it accepts only what it can sustain. Time-sensitive output such as a turn notification should be discarded once its validity window has passed rather than delivered late.

What does offline-first operation not protect? Anything requiring genuine coordination across counters. Routing people between counters based on global load cannot function during a network partition, so the system has to degrade to per-counter operation and resume routing afterwards. Local ledgers also move durability to the edge, so a terminal lost or destroyed before reconciling takes its unsynced events with it.