Hold the model fixed and lower the compute clock. If latency barely moves, you were not compute-bound. Then hold the compute clock and lower the memory clock. If latency moves roughly in proportion, you are memory-bound and now you know it rather than believe it. Most teams run these after a quarter of failed optimisation, which is the same information at a much worse price.
The arithmetic, so you can check it
The floor. Take the bytes that must cross the bus for one inference: the weights in the precision you actually run them, plus the activations you move. Divide by the memory bandwidth the part is specified for. That is a hard floor. Nothing you do to the arithmetic gets below it.
The share. Compare that floor against your measured latency. If the floor accounts for most of what you measure, you are memory-bound and every hour spent on faster kernels is an hour aimed at the part that was already idle. If it accounts for little, something else is taking the time, and preprocessing and postprocessing are the usual answer because they move bytes too and are routinely unmeasured.
What the target needs. Run it backwards: the bandwidth required to hit your target latency is the bytes divided by that target. If the number that comes out is not available on any part you would ship, the answer is fewer bytes, which means lower precision, a smaller model, or less activation traffic.
Specified bandwidth rather than achieved, which is usually 60 to 80 percent of it, so the real floor is higher than this reports. It also says nothing about the tail, which is set by contention for the bus rather than by your model and is a system problem living on a different layer entirely.
The part everyone gets wrong next
Accuracy, latency, memory footprint and power are not four independent targets. They are four views of one quantity: bytes across the bus. A bigger model is more bytes, lower latency is fewer bytes per inference, the footprint is literally the bytes, and the power is dominated by moving them. That is why integration ambushes people who tracked the four separately.
The reasoning is in your AI is memory-bound, not compute-bound.
Questions
Divide the bytes that must move for one inference by the part's memory bandwidth. If that figure is close to your measured latency, memory movement is your latency and there is nothing left for the arithmetic to explain. Confirm by lowering the compute clock, which should barely change latency, then the memory clock, which should change it roughly in proportion.
Because most models operate in the memory-bound region with the compute units already idle, waiting for data. Adding arithmetic throughput does not help when arithmetic was never what you were short of. The lever is memory bandwidth, or moving fewer bytes.
Roughly the size of the weights in whatever precision you actually run them, plus the activations, plus any cache traffic. A seven billion parameter model at eight bits is about seven gigabytes of weights alone, which at 200 GB/s is a floor of about 35 milliseconds before anything else happens.
Different memory system, different bandwidth, and different contention. A benchmark on a machine with bandwidth to spare and nothing competing for it says almost nothing about an edge part. Measure the whole pipeline on the real hardware while the rest of the system is running.