Ground Truth

What a Minute of Line Downtime Buys You in Architecture

Mostafa DhouibMostafa Dhouib··7 min read
The short answer

Downtime on an automated line runs from hundreds of thousands to millions of dollars an hour. Convert that to a per-minute figure and it becomes an engineering budget, which changes what is worth building. The interesting consequence is not that you should spend more. It is that the false-positive and false-negative costs are wildly asymmetric, and almost nobody sets the operating point from the plant's economics.

What a Minute of Line Downtime Buys You in Architecture

The short answer. Downtime on an automated line runs from hundreds of thousands to millions of dollars an hour depending on the process. Divide by sixty and you have an engineering budget per minute avoided, which makes a lot of previously unjustifiable work obviously worth doing. But the more useful consequence is that a false positive and a false negative cost different amounts, often by an order of magnitude, and a system tuned to a symmetric metric is tuned to the wrong thing.

Most inspection and maintenance systems are tuned to maximise accuracy, or F1, or whatever the framework defaulted to. Then they are deployed onto a line where being wrong in one direction stops production and being wrong in the other ships bad product, and those two do not cost the same.

The plant's economics were available the whole time. They were just never written into the system.

Start by converting the number

The first move takes an hour and changes every argument that follows.

Get the real figure for your line: what an hour of unplanned downtime costs, including lost throughput, restart, scrap during ramp, labour, and any contractual consequence of a missed commitment. Then divide.

Cost of an hour of unplanned downtimehundreds of thousands to millionslost throughput, restart, scrap during ramp, labour, contractual consequence
Divided by sixtya budget per minute avoided
A change that avoids ten minutes a monthnow has a number attachedand can be weighed against the two weeks it costs to build
Most teams have never done this arithmetic, so every reliability investment is argued on intuition against a feature roadmap that has revenue numbers on it.
FigureThe first move takes an hour and changes every argument that follows: convert the plant's downtime cost into an engineering budget per minute avoided.

The point of the per-minute figure is that it converts engineering decisions into comparisons. A change that reliably avoids ten minutes of stoppage a month has a number attached, and it can be weighed against the two weeks it costs to build. Most teams have never done this arithmetic, so every reliability investment is argued on intuition against a feature roadmap that has revenue numbers on it.

The asymmetry is the real finding

Now put the two error types side by side, because they are not symmetric and the gap is usually large.

A false positive flags a good part as defective, or a healthy machine as failing. On an inspection system, that stops the line. On a maintenance system, it triggers an intervention on equipment that did not need it, consuming a maintenance window and the crew's trust.

A false negative misses a real defect or a real developing fault. Bad product ships, or the machine runs to failure, which is an unplanned stop rather than a planned one, plus whatever the failure damaged.

False positive
Good part flagged defective, healthy machine flagged failing
The line stops, or a maintenance window is consumed
Immediate and visible
And it spends the crew's trust
False negative
Real defect missed, developing fault missed
Bad product ships, or the machine runs to failure
An unplanned stop rather than a planned one
Plus whatever the failure damaged
The threshold is the decision, not the model's reported number, and it should be derived from these two costs rather than from the point that maximises a symmetric metric.
FigureThe two error types do not cost the same, and the gap is usually large. Which one dominates is a plant decision, not a data science one.

The ratio between those two is the operating point, and it is a plant decision rather than a data science one. On a line where a missed defect reaches a customer, the false negative may dominate by a wide margin. On a line where a stop costs a fortune and defects are caught downstream anyway, the false positive dominates.

The number your model reports is not the decision. The threshold is the decision, and it should be derived from those two costs rather than from the point that maximises a symmetric metric.

The failure mode nobody records

There is a third cost that does not appear in either column and it destroys more systems than both.

A system that stops the line too often gets overridden. Then ignored. Then switched off, or left running with its output disconnected from anything.

An inspection or maintenance system nobody trusts is worse than none, because it carries cost without delivering protection, and the day it stopped being believed is never recorded anywhere. There is no incident, no ticket, no metric. Six months later somebody asks why the alerts are not acted on and the answer is institutional rather than technical.

This is why false-alarm reduction is often the highest-value work available on a deployed system, ahead of any accuracy improvement. Condition monitoring across an 87-unit fleet where false alarms fell 41 percent bought more than the equivalent gain in detection would have, because it bought the system back into use.

Where the false alarms actually come from

Worth naming, because the instinct is to blame the model and the cause is usually below it.

Regime change. The same healthy machine produces different vibration and current signatures at different loads and speeds. Judged against a global baseline, normal operation flags as anomalous every time the regime changes. This is the single largest source of false alarms in condition monitoring, and the fix is a per-regime baseline rather than a better classifier.

Single-sample alarms. A threshold crossed once is noise. A threshold crossed and sustained is a state. Alarming on sustained state rather than on a single sample removes a large share of alarms without touching sensitivity to real faults.

Line variance the system never saw. The development set captured the line on a good day. Production brings shift-to-shift lighting changes, material lot variation, and wear. The system degrades as the line drifts away from what it was built against, and it presents as increasing false alarms rather than as an error.

Sensor and mounting problems. A loosening accelerometer mount changes the signature. So does a camera that has drifted, a light that has aged, a lens that is dirty. These produce anomalies faithfully and the anomaly is real; it is just not about the machine.

Regime changethe same healthy machine reads differently at different loads and speeds. Judged against a global baseline, normal operation flags every time the regime changes
Single-sample alarmsa threshold crossed once is noise. Crossed and sustained is a state
Line variance never seenthe development set captured a good day. Production brings shift lighting, lot variation, wear
Sensor and mounting problemsa loosening mount, a drifted camera, an aged lamp. The anomaly is real, it is just not about the machine
The modelthe layer everyone starts with, and rarely the cause
A system that stops the line too often gets overridden, then ignored, then switched off. The day it stopped being believed is never recorded anywhere.
FigureWhere false alarms actually come from. Three of the four are below the model, which is the general shape of industrial work: the deciding decisions are usually not the inference.

Three of those four are below the model, which is the general shape of industrial work: the deciding decisions are usually not the inference.

What the budget justifies

With a per-minute figure in hand, here is what typically becomes obviously worth building and usually is not.

Per-regime baselines. More engineering than a global threshold, and it removes the dominant false-alarm source.

Sustained-state alarming with an explicit dwell time. Cheap, and it is the highest ratio of alarms removed to sensitivity lost of anything on this list.

Drift monitoring on the inputs, not just the outputs. The line changes continuously. A system that watches whether today's inputs resemble the ones it was built against catches degradation before it shows up as false alarms.

Sensor health as a first-class signal. Mount integrity, camera focus, illumination level, calibration age. Distinguishing "the machine is anomalous" from "the sensor is anomalous" is the difference between an actionable alert and an eroded one.

A recorded operating point with the costs written next to it. So that when someone retunes the threshold in eighteen months, the reasoning is in the repository rather than in a departed engineer's memory.

The question for your next review

"What does a false positive cost us, what does a false negative cost us, and where is that written down in the system?"

If the first two answers exist and the third is nowhere, the threshold in production is a default from a framework, and it is deciding a question with real money attached using no information about your plant.

That gap is typical of industrial automation programs, and closing it is usually a day of work with a return that dwarfs the next accuracy improvement.

FAQ

How much does production line downtime cost? It runs from hundreds of thousands to millions of dollars an hour depending on the process, including lost throughput, restart, scrap during ramp, labour and contractual consequences. Converting it to a per-minute figure turns reliability engineering decisions into comparisons you can actually make.

How should I set the threshold on an inspection or maintenance system? From the relative cost of a false positive and a false negative at your plant, not from the point that maximises a symmetric metric. Those two costs frequently differ by an order of magnitude, and the direction depends on whether a stop or an escaped defect is worse for your line.

Why do condition monitoring systems produce so many false alarms? Mostly for reasons below the model: the same healthy machine produces different signatures at different loads and speeds, alarms fire on single samples rather than sustained state, the line has drifted from the development conditions, and sensor or mounting problems produce genuine anomalies that are not about the machine.

What is the real cost of false alarms? Beyond the stoppages, the system gets overridden, then ignored, then switched off. A system nobody trusts carries cost without delivering protection, and the day it stopped being believed is never recorded anywhere, so nobody can point to when it happened.

What is the highest-return work on a deployed industrial system? Usually false-alarm reduction rather than accuracy improvement, because it buys the system back into use. Per-regime baselines and sustained-state alarming with an explicit dwell time remove a large share of alarms with little loss of sensitivity to real faults.

Free worksheet
The Pre-Rebuild Diagnostic

The four boring checks and the data-distribution slice, ending in a rebuild-or-repair verdict with the evidence for it. An afternoon of work, and it is designed to be carried into the meeting where somebody is proposing six months.

One email, the resource, and nothing else unless you reply.
Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program