When a robotics system fails in the field, it does not produce a worse benchmark. Depending on the application it produces a collision, a stuck robot, a safety incident, or a deployment nobody trusts to run unattended. An untrusted robot is not deployed at all, and the programme has spent its budget on something that will not ship.
A robot's perception is one stage of a control loop, and a control loop runs on a clock.
A robot that acts on what it senses is running a loop. Sensors capture the world, perception turns raw data into a description of what is there, state estimation combines that description with what the robot knows about its own motion, control decides what to do next, and actuators do it. The world changes in response and the loop runs again. It runs at a fixed period, and each pass has to finish before the next one is due.
That clock changes what good perception means. On a benchmark a perception model is scored on accuracy alone, and a slower, larger model that scores higher counts as a better model. In the loop, a result that arrives after the deadline is worse than useless, because the controller either acts on stale information or acts on nothing. Averages hide this. A stack that uses most of its budget on a typical cycle and occasionally overruns will occasionally do the wrong thing, and in a closed loop one wrong action shapes the next cycle's input, so errors can feed on themselves. Engineers therefore budget for the worst case and reserve headroom for jitter, the variation from one cycle to the next in how long each stage takes.
The second reality is the gap between the data a system is developed on and the world it meets. Simulators and benchmark datasets contain the conditions someone thought to include. The field contains glare, dust, reflections, unusual objects, and sensors that drift with temperature and age and differ slightly from one unit to the next. A model's score on its test set says how well it handles the test set. It says little about how far performance falls on conditions the test set lacks, and that fall cannot be predicted from the score. Robustness to those conditions usually costs computation, which is exactly what the deadline is short of, so the two constraints have to be traded against each other by design.
Two realities shaped everything, and neither appears on a leaderboard. The first is the distribution. The field is full of conditions the simulator never modelled, so a high offline score does not transfer, and how far it falls short cannot be predicted from the score itself. The second is the clock. Perception has to finish inside the control loop period on every cycle, with headroom left for jitter, and a stack that fits on average but overruns now and then does the wrong thing now and then, which in a closed loop is the whole problem. The two fight each other, because robustness usually costs computation and computation is exactly what the deadline lacks.
Robots that score well offline fail in the field in two predictable ways.
The first is distribution. Development data covers what someone thought to collect or simulate, and the field always contains more. A perception stack can perform superbly on its test set and degrade sharply on lighting, surfaces or objects it has never seen, and nothing in the offline score warns of it. The failure shows up as a robot that hesitates, misjudges a distance, or stops for something that is not there.
The second is time. Perception is usually developed and evaluated off the robot, where nothing has a deadline. Once it runs on the robot, sharing compute with everything else, its latency varies, and the tail of that variation is what matters. Overruns are rare, so they rarely show in a demonstration, and they are exactly what makes a robot untrustworthy unattended. Both failures tend to belong to nobody. The perception team owns the score, the controls team owns the loop, and the gap between them is where the robot fails.
The reflex, and the fix.
Optimise offline accuracy
It is the standard metric, and it improves cleanly with more model and more training, which makes it satisfying to work on.
A higher score survives neither the gap between simulation and the field nor the loop's deadline.
Build for the real environment and its timing
More work on robustness, sensor reality, latency and integration, and possibly a lower offline score.
A system that holds on the real distribution, inside the loop's deadline.
Make it work on the robot, in the world. The leaderboard is a proxy, and here it was the wrong one.
Perception and control as one real-time loop
Perception feeds control on a clock, and the loop closes through the physical world. It was built and tested with hardware in the loop, because that is the only place the real failure cases appear. A simulator cannot produce the failure a particular sensor shows at a particular temperature if nobody modelled it.
The timing budget set before anything was built
The loop period was divided explicitly across sensing, perception and inference, state estimation, control and actuation, with headroom reserved for jitter. This is an architectural input. Get it wrong and no amount of later optimisation recovers it.
Sensor reality as a first-class concern
Calibration, unit-to-unit variation and degradation over time, and what the system does as the real sensors drift away from the assumptions the software was built on. This is a hardware property surfacing as a software failure, which is why it is so consistently nobody's ticket.
What each choice cost, and what it bought.
AAllocating the timing budget first
Perception could have been built for accuracy and made fast afterwards, which is the usual order. The loop period was divided across sensing, perception and inference, state estimation, control and actuation before any of them was built, with headroom reserved for jitter. That limited how much model perception could afford from the outset. It also meant the stack could not be discovered too slow at integration, a failure that later optimisation rarely recovers from.
BTesting with hardware in the loop
Simulation is cheaper, faster and repeatable. It is also blind to failures nobody modelled, such as how a particular sensor behaves at a particular temperature. Building and testing with real hardware in the loop was slower and harder to automate. It is the only place those failures appear, and finding them on a bench costs far less than finding them in the field.
CMaking sensor reality somebody's job
Calibration, unit-to-unit variation and degradation over time sit between hardware and software, so they tend to belong to nobody. Treating them as a first-class concern meant designing for what the system does as real sensors drift from the assumptions it was built on. That took effort away from improving the model. It addressed a class of field failure that a better model would never have touched.
DAccepting a possibly lower offline score
Work spent on robustness, latency and integration is work not spent raising the benchmark number, and the offline score may well be lower as a result. That cost is visible, and it is easy to read as going backwards. The return is a system that holds on the real distribution, inside the deadline, which is the only performance the robot's operators will ever experience.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
A high benchmark score that survives neither the field nor the clock, and looks like progress right up until deployment.
Perception and action that hold reliably in the field, inside the control loop's timing, including the failure cases that only appeared with hardware in the loop.
What it owns, and what it hands to a person.
It holds on the conditions and failure modes it was built and tested against. A genuinely new scenario can still defeat perception. It depends on sensor calibration and on the operating envelope it was designed for, and the gap between simulation and the field has been narrowed, not closed. Anyone claiming otherwise is describing a system that has not met the field yet.
The deciding constraints were a hard timing budget and real sensor behaviour. Both sit below the model, and neither is something a perception specialist would have owned. The deadline is an architectural input, not a performance concern to deal with after the fact. The discipline carries to any system that closes a loop through the physical world, including the edge autonomy and hydraulic control work on this site.
A perception result that arrives after the control deadline is a wrong result. Judge perception in the loop, on the clock, and treat the leaderboard as a proxy.
Budget the loop period across every stage before building any of them, and reserve headroom for jitter. Timing problems found at integration are rarely fixable by optimisation.
An offline score says how well a model handles its test set. It cannot tell you how far performance falls on conditions the test set lacks.
Sensor calibration, unit variation and drift are hardware properties that show up as software failures. Give them an owner.
Hardware-in-the-loop testing is slower than simulation and finds the failures simulation cannot, because nobody modelled them.