Perception Accuracy Is Not Control Safety

Mostafa Dhouib·September 7, 2026·6 min read

Perception Accuracy Is Not Control Safety

The short answer. Accuracy is an average over a dataset. Safety is a property of the worst case in a specific physical situation. A detector at 99 percent can be perfectly safe or entirely unsafe depending on which one percent it misses, whether those misses cluster, what the system does when uncertain, and whether the safe behaviour is enforced below the layer that can be wrong. None of those four is visible in the accuracy number.

A perception metric is offered as evidence of safety more often than anything else in robotics, and it cannot carry that weight. Not because the number is inflated, but because it is answering a different question.

Four reasons the number does not transfer

One: errors are not interchangeable

Accuracy weights every mistake equally. Physics does not.

Missing a wall is different from missing a person. Missing an object outside the swept volume is different from missing one inside it. Missing a stationary object you are approaching slowly is different from missing one on a collision course.

A detector optimised for aggregate accuracy will happily trade a rare, consequential error for several common, harmless ones, because that trade improves the metric. Nothing about the training objective knows the difference, and nothing in the reported number reveals which trade was made.

The safety-relevant measurement is per-class, per-region, per-relative-velocity recall, and specifically recall on the categories where a miss is unrecoverable. That is a different table from the one usually presented, and it is usually worse.

Two: errors cluster, and clustering is what hurts

An independent one percent error rate is survivable, because consecutive frames give you many chances and a single missed frame changes little.

Real perception errors are not independent. They cluster on conditions: a particular lighting angle, a reflective surface, a pose the training set underrepresented, a distance where the sensor resolution falls off. Under those conditions the failure rate is not one percent, it is sustained, for as long as the condition persists.

That is the difference between a dropped frame and a blind interval, and only the second one causes an incident. An aggregate number cannot distinguish them, because it has already averaged over the axis where they differ.

The measurement that matters is the longest consecutive-miss interval per condition, not the mean rate. Almost nobody computes it, and it is straightforward to extract from data you already have.

Three: confidence is not calibrated to risk

A detector's confidence output is a number between zero and one, produced by a softmax that was never asked to mean anything physical.

Two failure directions follow. Confidently wrong: the detector reports 0.98 on an object that is not there, or on a misclassification, and it does so most readily on the out-of-distribution inputs it has never seen, which are precisely the interesting ones. And uncertain when it should not be: legitimate detections that fall below a threshold and get discarded, producing exactly the silent absence a controller cannot interpret.

Neither shows up in accuracy, and both determine what happens at the moment a decision is made.

Four: safe behaviour must be enforced below the perception stack

The structural point, and the one that actually resolves the problem.

If safety depends on perception being right, then safety is bounded by perception accuracy, and perception accuracy is a statistical property that cannot be driven to certainty.

So safety cannot be a downstream consequence of a good detector. It has to be a property enforced somewhere the perception stack cannot override.

In practice that means some combination of:

Independent sensing for the safety function. A physical bumper, a safety-rated scanner, a light curtain, a force threshold. Something whose failure mode is uncorrelated with the perception stack's, for the same reason a verifier must be independent of the thing it verifies.

Envelope limits enforced in the controller. Maximum velocity in defined zones, maximum force, geometric no-go regions, enforced regardless of what perception reports.

Fail-closed on absence and on uncertainty. The defined behaviour when perception is silent or unsure is to slow or hold, not to proceed on the last known state.

Validation against real firmware, not against an interface document. What the controller actually does with a limit command, on the actual firmware, at the actual timing. Those two diverge, and the divergence is found on the robot rather than in the specification.

What to present instead of an accuracy number

When someone asks whether the system is safe, these five answer the question that was actually asked.

Recall on the categories where a miss is unrecoverable, broken out by distance, relative velocity, and region of the workspace.

The longest consecutive-miss interval observed, per condition, and what the controller does during an interval of that length.

The behaviour on absence and on low confidence, stated as a behaviour rather than as a threshold.

What enforces the safety envelope, and whether the perception stack can influence it. If it can, the envelope is advisory.

The evidence that the enforcement works on the real controller, at real timing, under load.

The framing that helps

Accuracy answers: how often is the system right?

Safety answers: what happens when it is wrong, and is that bounded?

Those are different questions with different evidence, and improving the first does not answer the second. A team can spend a year moving a detector from 97 to 99 percent and change nothing about the answer to the second question, because the answer to the second question does not live in the detector at all.

That is the general shape of robotics and autonomous systems work: the safety architecture is not a feature of the perception program, it is the precondition that lets the perception program be useful, and it is enforced somewhere else entirely.

FAQ

Does high perception accuracy make a robot safe? No. Accuracy is an average over a dataset, while safety depends on which errors occur, whether they cluster on specific conditions, what happens under uncertainty, and whether safe behaviour is enforced below the perception stack. None of those is visible in an accuracy number.

Why does error clustering matter more than error rate? Because an independent one percent error rate means dropped frames, which consecutive observations recover from, while clustered errors mean a sustained blind interval under a particular condition. Only the second causes incidents, and an aggregate rate has already averaged over the axis that distinguishes them.

Can detector confidence be used as a safety signal? Not directly. Confidence is a softmax output never calibrated to physical risk, and detectors are frequently most confident on out-of-distribution inputs, which are the interesting ones. Use it only after calibrating it against measured error, and never as the sole gate on a safety decision.

Where should robot safety be enforced? Somewhere the perception stack cannot override: independent safety-rated sensing whose failure mode is uncorrelated, envelope limits in the controller applied regardless of what perception reports, and fail-closed behaviour on absence or uncertainty. If perception can influence the envelope, the envelope is advisory.

What should be presented instead of an accuracy figure? Recall on unrecoverable-miss categories broken out by distance, velocity and region; the longest consecutive-miss interval per condition and what the controller does during one; the defined behaviour on absence and low confidence; what enforces the envelope; and evidence that enforcement holds on the real controller firmware under load.