Why Air-Gapped Changes Every Assumption in Your ML Pipeline

Mostafa Dhouib·September 7, 2026·7 min read

Why Air-Gapped Changes Every Assumption in Your ML Pipeline

The short answer. An air gap is not a deployment detail bolted on at the end. It breaks an assumption at every stage: training data arrives in batches with latency measured in weeks, artifact identity has to be established without a registry, you cannot query what version is running, evaluation cannot use production traffic, and failure discovery happens when someone walks back with a drive. Each of those has a fix, all the fixes are architectural, and every one is expensive to retrofit.

Teams treat the air gap as a packaging problem. The pipeline is built the normal way, and near the end someone is assigned to work out how the model gets across.

That framing is wrong in a way that costs a program its schedule, because the air gap does not change the last step. It changes every step.

Data collection: the loop closes in weeks, not minutes

In a connected system, production inputs flow back continuously. Hard cases accumulate, the distribution is observable, and retraining is a decision about when rather than whether you have the data.

Air-gapped, data comes back when someone physically carries it back. The loop closes in weeks or months, sometimes not at all, and what returns is a subset chosen by whoever was doing the carrying.

Three consequences worth designing around.

You must decide in advance what to keep. Storage on the platform is finite and the interesting cases are rare. A policy that keeps everything fills the disk with the ordinary; a policy that keeps only failures cannot tell you your false-negative rate, because a failure the system did not recognise is not recorded as one. Keep a stratified sample plus everything near the decision boundary, and decide it before the unit ships.

Your training distribution ages. Between collections, the field moves and your dataset does not. This is ordinary drift with the feedback loop removed, so the first sign of it is a degraded demonstration rather than a metric.

The returning data is not representative. It is biased by which units came back, which operators bothered, and which missions were run. Treat it as a convenience sample, and say so in the same sentence as any number computed from it.

Artifact identity: no registry, so hash everything

The most common cause of "it worked in testing" is that the artifact serving production is not the artifact you evaluated. Connected systems paper over this with a registry: the model has an id, the id is queryable, and you can check.

Across an air gap the chain of custody is a person, a drive, and a procedure. Every one of those is a place where a version can diverge without anyone lying about anything.

The replacement is mechanical and it costs almost nothing to build early:

Hash the artifact, not the version string. Record the hash at build, at packaging, at transfer, and at load on the target. Four points, four recorded values, compared automatically.

Make the running system able to state its own identity locally. It cannot phone home, so it must be able to display or log a hash on demand, in a form an operator can read out.

Version the whole pipeline, not just the weights. The preprocessing, the normalisation constants, the vocabulary, the thresholds, the configuration. A model that matches and a preprocessing step that does not produces exactly the same symptom as a wrong model, and takes far longer to find.

Evaluation: your test set is the whole world

Connected systems evaluate continuously against live traffic. Air-gapped, whatever you validated before shipping is the entire evidence base until someone returns.

That makes two ordinary practices load-bearing rather than optional.

Split by group, never at random. If the same operators, the same units, or the same conditions appear on both sides of the split, your number measures internal consistency and you will not find out otherwise for months. Hold out whole units, whole operators, whole sites, whichever will be new in the field.

Report by slice, and treat the worst slice with real volume as the decision. One aggregate number over a mixed set hides the condition where it fails, and in the field you meet the conditions rather than the average.

The cost of getting this wrong is asymmetric here in a way it is not in a web service. A connected system that ships with an inflated number finds out in a week. An air-gapped system finds out at the demonstration.

Observability: decide the black box before you ship

You cannot add a log line to a fielded unit. Whatever the system records is fixed at ship time, and it is the only account you will ever have of what happened.

Four things worth recording that teams routinely omit:

Inputs near the decision boundary, because those are the cases that teach you something.

Every time a fallback or default path was taken. These return silently by design, which is what makes them invisible and why the absence of the counter is itself a finding.

Every refusal. For a fail-closed system, the refusal rate is the primary health signal. A system refusing far more than expected is degraded even though nothing errored.

The artifact hash and the configuration in force, written into the same record, so an anomalous trace can be attributed to a build without anyone having to reconstruct what was deployed that week.

Dependencies: the environment must be reproducible offline

The mundane one that consumes the most calendar time.

No package index, no model hub, no container registry, no certificate revocation check, no time server. Every one of those is a network call some library makes without telling you, and each fails differently on a machine with no route out. Some hang until a timeout, which is worse than failing, because a hang at load time looks like a hardware problem.

The work is to build and validate the full environment offline from the start, not to discover the dependencies when the target machine is in front of you. That includes pinning everything by hash, vendoring what you cannot pin, and running the build once on a machine with the network physically disconnected. The last one finds things no audit of the manifest will.

The pattern underneath

Every item on this list is the same move: something that was a runtime capability becomes a design-time decision.

Data collection becomes a retention policy. Version checking becomes a hashing scheme. Evaluation becomes a split strategy. Debugging becomes a recording plan. Dependency resolution becomes a vendored environment.

That is why an air gap cannot be handled at the end. Each of those decisions has to be made before the thing it governs is built, and a team that defers them is not deferring work but choosing the defaults, which are all the connected ones.

This is the shape of most defense and uncrewed systems work, where the air gap is a requirement rather than a circumstance. It applies equally to any system deployed somewhere it cannot be reached: a device in a hospital, a controller in a substation, a unit in a vehicle.

FAQ

Is an air gap just a deployment constraint? No. It invalidates an assumption at every pipeline stage: data collection becomes batched and biased, artifact identity has no registry to lean on, evaluation cannot use production traffic, observability is fixed at ship time, and dependency resolution has no network. Each fix is architectural and expensive to retrofit.

How do you verify which model is running on an air-gapped system? Hash the artifact and record it at build, packaging, transfer and load, comparing automatically. Version the whole pipeline including preprocessing and configuration, and make the running system able to state its own hash locally, since it cannot report it anywhere.

How should an air-gapped system be evaluated? Split by group rather than at random, holding out whole units, operators or sites that will be new in the field, and report per slice rather than as one number. Whatever you validated before shipping is the entire evidence base until someone physically returns with data.

What should an air-gapped system record? Inputs near the decision boundary, every fallback or default path taken, every refusal, and the artifact hash plus configuration in force. It cannot be changed after shipping, so the recording plan is a design decision rather than an operational one.

Why do offline dependencies cause so much delay? Because libraries make network calls nobody declared: package indexes, model hubs, certificate revocation, time servers. Some fail, some hang, and a hang at load time presents as a hardware fault. Building once on a physically disconnected machine finds what no manifest audit will.