Perception to Control: The Handoff Nobody Owns

Mostafa Dhouib·September 7, 2026·7 min read

Perception to Control: The Handoff Nobody Owns

The short answer. Perception delivers a detection. Control consumes a world state. Those are not the same object, and the conversion between them carries five properties: what the detection means, when it was true, in which coordinate frame, how uncertain it is, and what to do when it is absent. Perception specifies the first. Control assumes the rest. The robot's bad behaviour lives in the gap, and it belongs to neither team.

Two competent teams, two passing test suites, one robot that does something strange in a corner of the workspace.

The perception team reports 96 percent detection accuracy on their validation set. The controls team reports a stable, well-damped controller. Both statements are true. Neither team can explain the behaviour, and each investigation ends politely at the interface.

The five properties that cross the boundary

A detection message looks like a simple object. It carries far more than its fields.

One: what it means

"There is an obstacle at position P" underspecifies. Does P mean the nearest point of the obstacle, its centroid, or the centre of its bounding box? Does the message assert that the space is occupied, or that the detector believes it is occupied?

The difference between a centroid and a nearest point is the difference between a controller that clears an object and one that grazes it, and both teams can be entirely correct within their own definitions.

Two: when it was true

The property that produces the most confusing behaviour, because the symptom is intermittent and load-dependent.

A detection describes the world at capture time, not at publish time. Between those sits exposure, transfer, preprocessing, inference, post-processing, and message passing. If control treats the arrival timestamp as the observation time, every estimate is stale by that pipeline latency, and the error scales with relative velocity.

Worse, the latency is not constant. It grows under load, when the scene is busy, when another process contends for the accelerator. So the controller sees an error that appears when things get difficult, which is exactly when it matters and exactly when it is hardest to instrument.

The fix is cheap and structural: stamp at capture, carry the stamp through every stage, and have control compensate for the age of the observation rather than assume it is current.

Three: in which frame

Camera frame, robot base frame, world frame, and whatever the calibration says relates them. A transform that is slightly wrong produces a systematic offset that looks like a control problem: the robot consistently approaches slightly off, and tuning the controller partially compensates, which buries the real cause under a workaround.

The tell is that the error correlates with position in the workspace rather than with speed or load. That single diagnostic separates a calibration problem from a timing problem in about an hour, and it is worth running before either team starts changing anything.

Four: how uncertain it is

Perception systems produce confidence values. Control systems need uncertainty in units that mean something physically: this position is known to within so many millimetres.

A detector confidence of 0.9 is not a position tolerance, and converting one to the other requires a calibration nobody usually does. So the number is passed across, control uses it as though it were a tolerance, and the resulting behaviour has no principled relationship to the actual error.

The honest options are to calibrate confidence against measured positional error, or to carry an explicit covariance, or to state a worst-case tolerance and design for it. What does not work is passing a number whose units are undefined and letting the consumer interpret it.

Five: what absence means

The property that is nearly always unspecified, and the one with the worst failure mode.

No detection this frame means one of: nothing is there, something is there and was missed, the sensor is obstructed, the pipeline dropped a frame, or the process died. Those demand completely different responses, and a controller that treats all of them as "nothing is there" will drive into an obstacle it briefly failed to see.

The requirement is that perception distinguishes them: an explicit "observed, nothing present" is a different message from "no observation available." And control must have a defined behaviour for the second, which is usually to hold or slow rather than to proceed.

This is the same structural blindness that makes a silent node hard to detect on a fleet: absence is not a signal, so nothing collects it, so nothing reacts to it.

Why neither team finds it

Each team's tests are honest and each is scoped to their own component.

Perception validates on a dataset: images in, detections out, compared to labels. There is no latency in a dataset, no coordinate frame issue, and absence is a labelled negative rather than an ambiguous silence.

Control validates against a simulated or idealised input: perfect state, current, in the right frame, with known uncertainty. The controller is stable given that input and would be stable given any input with those properties.

Neither test contains the interface. The properties that fail are exactly the ones that only exist when the two are connected, which is why the failure reproduces only on the real robot, and why each team's investigation truthfully concludes their side is fine.

What to do

Write the interface contract, both sides separately, then compare. All five properties: meaning, timing, frame, uncertainty, absence semantics. Filled in without conferring, because conferring produces agreement while independent filling produces the disagreement you are looking for.

Instrument the boundary rather than the components. Log what perception published, with capture and publish timestamps, and what control consumed, with its consumption time. The gap between those three numbers is a diagnostic nobody has until they build it.

Test the pair, not the parts. A test that injects a known object at a known position and checks where the robot believes it is exercises the whole chain, including the transform and the timing. That single test finds more than either team's suite.

Give the boundary an owner. One party accountable for the agreement, with authority to change either side. Not perception, not controls, and not a weekly sync between them.

This is the defining structure of robotics and autonomous systems work: the components are well understood, the specialists are strong, and the failures concentrate in a seam that the org chart does not assign to anyone.

FAQ

Why does a robot behave badly when perception and control both test clean? Because neither test contains the interface. Perception validates on a dataset with no latency, frame, or absence ambiguity; control validates against idealised state. The properties that fail exist only when the two are connected, which is why the failure appears only on the real robot.

What causes stale perception data in a control loop? Treating the arrival time as the observation time. A detection describes the world at capture, and exposure, transfer, inference, and message passing sit in between. The lag grows under load, so the error appears exactly when the scene is busy and matters most.

How do you tell a calibration error from a timing error? By what the error correlates with. A transform problem produces an offset that varies with position in the workspace; a timing problem produces an error that scales with relative velocity and load. Checking that correlation takes about an hour and settles which team should be looking.

Can detector confidence be used as a position tolerance? Not without calibration. A confidence of 0.9 has no defined relationship to millimetres of positional error. Either calibrate confidence against measured error, carry an explicit covariance, or state a worst-case tolerance and design for it.

What should a controller do when no detection arrives? Distinguish observed-nothing-present from no-observation-available, which requires perception to send different messages for the two. For the second, the defined behaviour is normally to hold or slow rather than to proceed, because a missed detection and an empty scene look identical otherwise.