Systems You Cannot Take Offline to Fix

Mostafa Dhouib·September 7, 2026·7 min read

Systems You Cannot Take Offline to Fix

The short answer. The default engineering loop is change something, run it, see what happened. On critical infrastructure that loop is unavailable: you get scheduled windows measured in hours, months apart, and everything between them has to be done without interrupting service. What replaces it is a different method: passive instrumentation as the primary diagnostic, shadow execution as the primary validation, one variable per window, and every change designed so that abandoning it halfway leaves a working system.

Grid, substation, plant, and process control share a property that invalidates most software practice: the system is doing something that must not stop.

That is not a constraint you work around. It changes the method.

What you lose

Worth listing explicitly, because each item is a habit rather than a tool, and habits are harder to notice.

Reproduce and iterate. The core debugging loop is unavailable. You cannot provoke the fault to study it.

Roll back after the fact. Sometimes you can, sometimes the change has already had a physical consequence. A control action that moved equipment does not un-move it.

Test in an identical environment. The estate is generational: equipment installed decades apart, from vendors who no longer support it. A test rig is a model of one part of it, and the parts that differ are the ones that matter.

Fail and learn. In most software a failed deploy is a bad afternoon. Here it can be an outage across critical infrastructure, a safety incident, or a compliance finding.

What replaces them

Four practices, and the striking thing is that all four are things every system should do and most do not. Critical infrastructure just removes the option of skipping them.

Passive instrumentation as the primary diagnostic

If you cannot provoke the fault, you have to be recording when it happens on its own.

This inverts the usual order. Instrumentation is normally added after a problem appears, to investigate it. Here it must exist before, because the event you need to see may occur twice a year and you get no say in when.

Tap the links and record every transaction with timing. On a control network this costs nothing in availability and answers questions nobody has asked yet.

Record state transitions, not just values. A value sampled every thirty seconds misses the transient. The transition is the event.

Capture the moments around anomalies, not only the anomaly. A ring buffer that retains the preceding minutes when something trips is worth more than any dashboard, because causes precede effects and dashboards are aggregates.

Count every fallback and every default. These return silently by design, and on a system you cannot interrogate, silence is the hardest thing to reconstruct after the fact.

Shadow execution as the primary validation

Run the new logic alongside the live one. Same inputs, same timing, writing nowhere.

You get production conditions with no production risk, and more importantly you get a comparison: what the new logic would have done against what the live system did, on real events, for as long as you care to run it.

For a control system this is often the only way to build confidence before a window, and it converts the window from an experiment into a deployment of something already evidenced.

The discipline that makes it work: decide in advance what agreement rate would satisfy you, and what disagreement would stop you. Otherwise you accumulate months of shadow data and make the decision on the same intuition you would have used without it.

One variable per window

You get a small number of chances per year. Spend each one buying a decision.

Two changes in one window and an unexpected outcome cannot be attributed. That is not a small inefficiency, it is the loss of a window that will not come again for months.

Which means the planning question is not "what can we fit in this window" but "what will this window let us conclude?" Those produce different plans, and the second one is almost always the better use of the access.

Every change abandonable halfway

Design each modification so that stopping partway leaves a working system.

Additive before subtractive. Add the new path, run both, remove the old one in a later window. Never in the same one.

Feature-flagged at the point of effect, not at the point of deployment, so reverting does not require another deployment.

Config over code where the behaviour is likely to need tuning, so an adjustment does not consume a window.

The test: if the window ends unexpectedly at any point during the change, is the system in a state that can run until the next one? If the answer is no at any moment, the change needs restructuring.

The generational estate problem

One more property specific to this domain, worth naming because it defeats otherwise sound approaches.

The equipment spans decades. Modbus, DNP3, IEC 61850, and OPC-UA across vendors and eras, and the standard is a starting point rather than a guarantee. A device behaves as it actually implements the protocol, which is discoverable only against that device.

Code written from the specification works on the bench and fails against the substation. So the observed-behaviour record, one document per device family recording what each actually does, is the highest-value artifact in the program and it almost never exists.

The compounding difficulty is that the oldest equipment is both the most divergent and the least tolerant of unexpected traffic. A scan that a modern controller shrugs off can knock over a device from an era when nothing scanned anything.

The convergence seam

The failure mode growing fastest in energy and utilities work is not any single system. It is the boundary between operational and IT networks as they converge.

That seam spans two organisations with different threat models, different change cadences, and different definitions of uptime. The operational side measures uptime in years and changes things during scheduled outages. The IT side patches on a cycle and considers a monthly maintenance window generous.

Neither is wrong. But a boundary owned by two parties with incompatible operating assumptions is owned by nobody, and it is where the exposure concentrates. The fix is the same as any seam: write down what crosses it, what each side assumes, and put one name against the agreement.

FAQ

How do you debug a system that cannot be taken offline? By instrumenting passively before the fault occurs rather than provoking it afterwards: tap the links and record every transaction with timing, record state transitions rather than sampled values, retain the minutes preceding an anomaly, and count every fallback and default.

What is shadow execution and why does it matter here? Running new logic alongside the live system on the same inputs while writing nowhere, so you can compare what it would have done against what the live system did on real events. It converts a maintenance window from an experiment into the deployment of something already evidenced.

Why only change one thing per maintenance window? Because two changes with an unexpected outcome cannot be attributed, and the next window may be months away. The planning question should be what this window will let you conclude, not what can be fitted into it.

What does it mean for a change to be abandonable halfway? That stopping at any point leaves a system that can run until the next window. In practice: add the new path before removing the old one, flag at the point of effect rather than deployment, and prefer configuration over code where tuning is likely.

Why does protocol code written from the standard fail in the field? Because a utility estate spans decades and vendors, and each device behaves as it actually implements the protocol rather than as the specification describes. The oldest equipment is both the most divergent and the least tolerant of unexpected traffic, so a scan a modern controller ignores can disable it.