A robotics system that fails in the field does not produce a worse benchmark. Depending on the application it produces a collision, a stuck robot, a safety incident, or a deployment that cannot be trusted to run unattended, which means it is not deployed at all and the program has spent its budget on something that will not ship.
Two realities shape everything and neither appears on a leaderboard. The distribution: the field is full of conditions the simulator never modeled, so a high offline score does not transfer, and the amount by which it fails to transfer is not predictable from the score itself. The clock: perception is bounded by the control loop period every cycle and has to leave headroom for jitter, and a stack that fits the budget on average and exceeds it occasionally does the wrong thing occasionally, which in a closed loop is the whole problem. The two constraints fight each other, because robustness usually costs computation and computation is exactly what the deadline lacks.
The reflex, and the fix.
Optimize offline accuracy
The standard metric. Improves cleanly with more model and more training, which makes it satisfying to work on.
A higher score survives neither the gap between simulation and reality nor the loop's deadline.
Build for the real environment and its timing
More work on robustness, sensor reality, latency, and integration. Possibly a lower offline score.
A system that holds on the real distribution, inside the loop's deadline.
Make it work on the robot, in the world, not on the leaderboard.
Perception and control as one real-time loop
Perception feeds control on a clock, and the loop closes through the physical world. It was built and tested with hardware in the loop, because that is the only place the real failure cases appear. A simulator will not generate the failure a particular sensor produces at a particular temperature, because nobody modeled it.
The timing budget allocated before anything is built
The control loop period is divided explicitly across sensing, perception and inference, state estimation, control, and actuation, with headroom reserved for jitter. This is an architectural input, not a performance concern to check at the end, and getting it wrong is not recoverable by optimizing later.
Sensor reality as a first-class concern
Calibration, unit-to-unit variation, and degradation over time, and what the system does as the real sensors diverge from the assumption the software was built against. This is a hardware property surfacing as a software failure, which is why it is so consistently nobody's ticket.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
A high benchmark score that does not survive contact with the field or the clock, and that looks like progress right up until deployment.
Perception and action that hold reliably in the field, within the control loop's timing, including the failure cases that only showed up with hardware in the loop.
What it owns, and what it hands to a person.
It holds on the conditions and failure modes it was built and tested against. A genuinely novel scenario can still defeat perception, it depends on sensor calibration and the operating envelope it was designed for, and the gap between simulation and reality is narrowed rather than closed. Anyone claiming otherwise is describing a system that has not met the field yet.
The deciding constraints were a hard timing budget and real sensor behavior, both of which sit below the model and neither of which a perception specialist would have owned. The deadline is an architectural input, not a performance concern to address after the fact. The discipline carries to any system that closes a loop through the physical world, which includes the defense edge autonomy and the hydraulic control work on this site.