Where Your Inference Actually Runs

Mostafa Dhouib·10 min read

Where Your Inference Actually Runs

The short answer. Teams assume that choosing a region for a model service means inference happens in that region. Frequently it does not. Capacity is pooled and requests can be routed across regions to whichever cluster has room, and some providers' highest-throughput or lowest-cost endpoints state explicitly that they carry no residency guarantee at all. The storage region and the inference region are separate questions with separate answers. Establish both in writing, for your specific endpoint and deployment type, before any part of the architecture depends on them. Then design so the answer is allowed to change, because it will. Every specific term here must be re-verified against current provider documentation.

The conversation usually happens late. An architecture is built, a data protection assessment is being written, and someone asks where the model call goes. The team points at a configuration value naming a region. That value is often about something else.

The mechanism, without the vendor names

Running large models is capital-intensive and demand is spiky. Providers manage that by pooling capacity, and pooling is in tension with keeping each request inside a single geography. The ways this shows up are consistent enough to generalise.

Storage region and processing region are configured separately, if both are configurable at all. The region you selected may govern where account data, logs or stored artefacts live, while the compute serving your request is allocated from a different pool. Both facts can be true at once, and a dashboard showing one region is not a statement about the other.

Cross-region routing is a documented feature, not an incident. To improve availability and throughput under load, some services will serve a request from a cluster outside the region you nominated. It is usually described as a benefit, which it is, operationally. It is also a statement that the processing boundary is wider than the configured region.

Global or shared endpoints frequently disclaim residency explicitly. The cheapest and most scalable endpoints are often the ones with the broadest routing, and the documentation says so. A team that picked the endpoint for its price and throughput has, without noticing, accepted the routing terms attached to it.

Residency commitments, where they exist, attach to a specific deployment type. They are a property of a particular configuration, sometimes a particular contract, often at higher cost or lower capacity, and sometimes with a reduced feature set or delayed access to new models.

Zero data retention and data residency are different commitments. A provider may commit not to store your request content at all while still routing the computation anywhere. That is a meaningful privacy control and it is not a residency control. Teams conflate them constantly, in both directions.

Subprocessors matter. Some providers serve models they do not host themselves. The residency question then applies to whoever actually runs the inference, and you need the subprocessor list to even ask it properly.

I am not naming providers, regions, endpoint types or dates, on purpose. Every one of those specifics changes on a vendor's schedule, and an article that stated them would become confidently wrong. The mechanism is stable. The particulars must be verified against current documentation at the time you build.

How to establish the answer properly

"Establish" is the operative word. Not infer, not assume from a region dropdown, not take from a sales call.

Start from the documentation and record it. Find the current statement for the exact endpoint and deployment type you will use. Copy the relevant text into your design file with the date you read it. If the documentation is silent on inference location, that silence is your finding, and it is a finding you should write down.

Get it in the contract or a written commitment. Documentation is a description of current behaviour and can change without notice to you. A contractual term, a data processing agreement or a written answer from an account team is a commitment. For a regulated product, the difference between those two is the difference between a control and a hope.

Ask the questions that have unambiguous answers. Vague questions get reassuring answers. The useful set is narrow and specific. In which countries may inference execute for this endpoint and deployment type? Is cross-region routing possible, and can it be disabled? Which configuration pins processing to a geography, and what does it cost in money, capacity and feature availability? What exactly does the residency commitment cover: inference, prompt content, logs, caches, evaluation data, abuse-monitoring copies? Who are the subprocessors that may execute inference, and where are they? How will I be notified if any of this changes, and with how much notice? Can you state the commitment in the agreement rather than only in documentation?

Ask what happens under failure. Residency guarantees that hold in normal operation sometimes bend during a regional outage, when the choice is between serving from elsewhere and not serving. Find out which way the service is designed to fail, because for a regulated workload the correct answer may be to fail closed and return an error.

Verify rather than trust where you can. Some services return the serving region or an equivalent identifier in response metadata. Log it on every call and alert on anything unexpected. Where that is not available, accept that you are relying on a commitment rather than an observation, and say so in your assessment rather than implying evidence you do not have.

Do this before the architecture depends on it. The whole point. Establishing residency after you have built around an assumption converts a design question into a rebuild. This is the same discipline as proving data access before you build, applied to processing location rather than to availability of the data.

Designing so the answer can change

Even a well-established answer has a shelf life. Terms change, endpoints are replaced, you enter a new market with a different regime, your provider adds a subprocessor. Design for revision.

Put every model call behind one internal interface. One adapter per provider, with the application depending on your interface and not theirs. This is the standard pattern described in one adapter per provider, and residency is one of the strongest reasons for it. If switching provider or endpoint touches one module, a change of terms is a project. If it touches forty call sites, it is a crisis.

Make the endpoint, region and deployment type configuration, with a named validated combination. Not constants scattered through code. A single configuration object you can review, assert on in tests, and change under change control.

Classify data before it reaches the call. Decide per field, not per product, which data may cross a boundary. Often the regulated subset is small. Pseudonymising or omitting identifiers before the call can move the majority of your traffic out of the restricted category entirely and leave a much smaller problem to solve expensively. This is the same logic as deciding the system of record per datum rather than for the whole product.

Keep the model-dependent step thin. If a residency constraint forces you onto a different model, a smaller model or a self-hosted one, the damage is contained to the extent that everything else in the pipeline is deterministic and provider-independent.

Have a degraded path. What the product does when no compliant inference option is available is a design decision. Queue the work. Fall back to a non-model path. Disable the feature for affected jurisdictions. Decide which, and build it, before you need it.

Review the terms on a cadence and record each review. Put it in the same recurring slot as your model lifecycle review, as described in model retirement as a regulatory event. An assessment written once and never revisited is evidence of a past state.

The honest limit

Residency is one requirement among several, and it is not always the binding one. Depending on your regime, the controlling questions may be about lawful basis, transfer mechanisms, onward disclosure, retention or the rights of the data subject, and inference location may be a smaller part of that picture than the architecture debate suggests. Work out which requirement actually binds before optimising for this one.

I am not a lawyer and this is not legal advice. What counts as a transfer, which mechanisms are available to you and what your obligations are under the regime that applies to your product are questions for counsel who know your jurisdiction and your sector. The engineering point stands independently: you cannot satisfy a requirement you have not established, and you cannot establish this one by reading a region name in a console.

Finally, pinning inference to a geography has costs that are easy to understate. Fewer models, older models, less capacity, higher prices, slower access to improvements, and sometimes a worse product. That trade may be entirely correct for a regulated device. It should be made with the numbers in front of you, at the start, rather than discovered when someone asks the question nine weeks before an audit.

FAQ

Does choosing a region for a model service mean inference happens there? Often not. The region you configure may govern where account data, logs or stored artefacts live while the compute serving your request is allocated from a pooled capacity set that spans regions. Storage location and processing location are separate questions with separate answers, and a console showing one region is not a statement about the other.

What is cross-region routing and why does it matter? It is a documented capability where a request nominated for one region is served from a cluster elsewhere, to improve availability and throughput under load. Operationally it is a benefit. For a residency requirement it means the processing boundary is wider than the region you selected, and you need to know whether it can be disabled and at what cost.

Why are the cheapest endpoints often the riskiest for residency? Because the broad routing that makes them cheap and scalable is the same thing that removes the geographic guarantee, and the documentation for some global or shared endpoints states that no residency commitment applies. Teams select such an endpoint for price and throughput and inherit its routing terms without registering that they have made a residency decision.

What should I ask a provider? In which countries may inference execute for this exact endpoint and deployment type. Whether cross-region routing can be disabled and at what cost in money, capacity and features. Precisely what the commitment covers: inference, prompt content, logs, caches, abuse-monitoring copies. Who the subprocessors are and where they run. How changes are notified. And whether it can go in the agreement rather than only in documentation.

Is zero data retention the same as data residency? No, and conflating them is common. A provider can commit not to store your request content while still routing the computation to any region with capacity. Retention concerns whether content is kept, residency concerns where processing and storage occur. You may need both, and each has to be established separately for your specific endpoint.

How do I design so a residency answer can change? Put all model calls behind one internal interface with a single provider adapter, hold endpoint, region and deployment type as reviewable configuration with one named validated combination, classify data per field so only the genuinely restricted subset needs the expensive path, keep the model-dependent step thin, and build the degraded path for when no compliant option exists.