Edge or Cloud Inference: A Decision Table

Mostafa Dhouib·September 8, 2026·6 min read

Edge or Cloud Inference: A Decision Table

The short answer. Three questions decide it and none of them is latency. Must the system work when connectivity is absent? Can the data legally and contractually leave the premises? Can you reach the device afterwards to fix what you got wrong? Answer those and the choice is usually made. Latency and cost then tell you whether the answer is affordable, which is a different question from whether it is correct.

The three deciding questions

Must it work with no connectivity? Not "is connectivity usually available." Must it work during the outage. If a five-minute gap means the system stops doing something that must not stop, that is edge, and no amount of clever buffering changes it.

Can the data leave? Regulation, contract, or a customer's security posture. This is frequently decided outside engineering and discovered late, which is expensive because it invalidates an architecture rather than a component.

Can you reach it later? A device in a customer's facility, a vehicle, or a remote site has an update path that may be slow, unreliable, or absent. That determines how much you need to be right the first time, and it is the question people skip.

If all three point the same way, you are done. Where they conflict, the split-the-work section below is usually the answer rather than a compromise on one of them.

The decision table

Two rows deserve attention because they get inverted in most discussions.

Cost at scale runs the opposite way to intuition. Cloud inference is cheap to start and does not amortise: at ten times the volume the bill is ten times. Edge is expensive up front and roughly flat per unit afterwards. So a pilot comparison favours cloud and a mature-volume comparison often favours edge, and teams choose during the pilot.

Retry is the hidden dependency. Cloud gives you retry on timeout, escalation to a bigger model, logging for tomorrow, and a patch next week. All four vanish at the edge simultaneously, which means the same architecture delivers materially worse reliability there with no bug and no code change.

What decides the edge envelope

If the answer is edge, the constraint is fixed earlier than most teams expect, and by people not in the AI conversation.

The compute module and its memory bandwidth, the thermal design that sets sustained rather than peak power, and the platform firmware including how it schedules the interrupt your inference sits behind. All chosen before there is inference code, and all binding.

The arithmetic worth doing before parts are selected: model size in bytes, times how often it must cross the bus, against the part's memory bandwidth. That gives a latency floor no optimisation reaches past. If your requirement is below the floor, the requirement needs a different part rather than a better model.

The characteristic expensive failure is a model that works, on the target hardware, and misses the latency budget because of firmware scheduling. Across vendors that costs weeks establishing ownership; the fix is often an afternoon.

Splitting the work

Most real systems are neither. The useful split is by what must be true when the network is gone.

At the edge: anything on a control path, anything that must produce a bounded decision now, and anything whose input cannot leave. Keep the output space enumerable so a wrong result is the wrong member of a known set rather than an arbitrary one.

In the cloud: anything that can wait, anything that benefits from a larger model, aggregate analysis across units, and the training loop.

Between them: a queue that degrades honestly. When connectivity returns, the edge sends what it recorded; when it is gone, the edge continues without it and the system knows which mode it is in.

The design test for the split: if the link is down for a day, what stops working, and is that acceptable? If the honest answer is that the product stops, the boundary is in the wrong place.

What to decide before choosing

The three deciding questions, written down with a name against each. The data-residency one especially, since it is answered outside engineering.

The latency floor arithmetic, before parts are selected.

The behaviour on absence and uncertainty, if edge. With no larger model to escalate to and no reviewer downstream, the answer is usually a refusal, enforced where the rest of the system cannot bypass it.

The recording plan, if edge. Whatever the unit writes down is fixed at ship time and is the only account you will ever have of a failure.

The cost at target volume rather than pilot volume, for both options, because that comparison frequently reverses.

FAQ

Should inference run at the edge or in the cloud? Decide on three questions: whether the system must work with no connectivity, whether the data can legally and contractually leave, and whether you can reach the device later to fix mistakes. Latency and cost tell you whether the answer is affordable, not whether it is correct.

Is edge inference cheaper than cloud? At volume, usually. Cloud inference is per-request and does not amortise, so ten times the traffic is ten times the bill, while edge cost is largely fixed at manufacture and roughly flat per unit. A pilot-scale comparison favours cloud and a mature-volume one often does not, and teams decide during the pilot.

Why is edge reliability worse with the same architecture? Because the cloud was silently providing retry: retry on timeout, escalation to a larger model, logging for later, and patching. All four disappear together at the edge, so the single-attempt reliability becomes the actual reliability with no bug and no code change.

What sets the edge latency floor? Memory bandwidth, in most cases. Model bytes crossing the bus per inference against the part's bandwidth gives a floor no optimisation reaches past. That is fixed when the compute module is chosen, well before there is inference code.

How should a hybrid system be split? By what must be true when the network is gone. Control paths, bounded decisions needed now, and data that cannot leave go at the edge; anything that can wait, benefits from a bigger model, or aggregates across units goes to the cloud. Test it by asking what stops working during a day-long outage.