Bound the Concurrency or Your Latency Problem Becomes a Refusal

Mostafa Dhouib·7 min read

Bound the Concurrency or Your Latency Problem Becomes a Refusal

The short answer. When a request has several independent outbound calls to make, running them in sequence makes the latency the sum and running them concurrently makes it roughly the slowest one. That is the right instinct and it is incomplete, because unbounded concurrency has a different failure mode from slowness. A wide request with no ceiling opens as many connections as it has work items, and under several simultaneous users that exhausts a connection pool, trips a provider rate limit, or saturates an event loop. The result is refusals rather than delays, arriving in bursts exactly when traffic is highest. A ceiling converts that into a queue, and a queue is a slower response that still completes.

Serialised outbound calls are one of the easiest wins in a slow request. Five calls at four hundred milliseconds each is two seconds if you wait for each in turn and about four hundred milliseconds if you do not.

So the fix is obvious, the change is small, and the improvement is immediate. The part that gets skipped is the ceiling, because at the moment of the change there is no evidence it is needed.

What unbounded actually means

Under light load, concurrency is free. One user, five calls, five connections. Nothing notices.

The arithmetic changes with simultaneous users, and it changes multiplicatively. Twenty users, each making a request that fans out to forty candidates, is eight hundred outbound calls in flight. Nothing in your code says eight hundred. The code says "run these concurrently" and the number comes from the product of two things neither of which is a constant.

Three ceilings are waiting at around that point.

Connection pools. Most clients hold a pool with a maximum size and a queue behind it. Exceed it and requests wait for a connection, which is fine, or time out waiting, which is not. The timeout is usually configured for a slow server rather than for a full pool, so it fires early and reports a failure that is actually congestion.

Provider rate limits. The provider has an opinion about how fast you may ask. Exceed it and you get refusals, and some providers respond to sustained overage with longer penalties. This failure is worse than it looks because it is a reputational state with the provider rather than a transient error.

The runtime itself. On a single-threaded runtime, a large number of concurrent handlers competing for the loop makes everything slower, including the handlers that were not fanning out. The degradation is not confined to the feature that caused it.

Why refusals are worse than delays

A slow request completes. The user waits, perhaps longer than ideal, and gets an answer. A refused request does not.

Three reasons the distinction matters more than the average latency number suggests.

Refusals cluster. They happen at peak, so the users who experience them are a large fraction of the users you have at that moment. A latency increase is spread across everyone. A refusal rate is concentrated.

Refusals trigger retries, which make it worse. A client that retries on failure converts one refused request into several, at exactly the moment the system is saturated. This is the classic shape of a congestion collapse and it is self-reinforcing.

Refusals look like a different problem. A connection timeout reports as a provider error, so the investigation goes to the provider, who reports that they were fine. The actual cause is on your side, in a pool you did not know was full.

The ceiling

A bounded fan-out runs at most N calls at once and queues the rest. Everything still happens, just not all at the same moment.

Under the same twenty-user load, with a ceiling of eight per request, you have at most one hundred and sixty in flight rather than eight hundred. The pool holds, the provider stays inside its limit, and each request takes a little longer because some of its work waited.

Choosing N is a question about your constraints rather than a tuning exercise. It should fall below whichever of these binds first: your connection pool size divided by expected concurrent requests, your provider's rate limit divided by expected request rate, or whatever your runtime handles without degrading unrelated work. Start from the binding constraint rather than from a round number.

There is also a second ceiling worth having: a global one, across requests, not just within them. A per-request limit of eight with unlimited simultaneous requests still reaches the provider's limit. A shared semaphore or a worker pool is what actually bounds the system.

Degrade rather than queue forever

A queue with no limit is a different way of failing. Work arrives faster than it drains, the queue grows, and latency climbs until requests time out anyway, having consumed resources the whole time.

So the ceiling needs a companion decision: what happens when the queue is long.

The useful answer in a search-shaped product is to return the best answer found within the budget. Fewer candidates were priced, the result is slightly worse, and the user got something. This is the same per-request budget that bounds cost, doing a second job.

Shedding load explicitly is better than letting a queue do it implicitly, because an explicit decision can choose what to drop. An implicit one drops whatever was unlucky, which is often the request that had already done the most work.

Where this generalises

Anywhere one unit of work expands into several outbound operations against something with a limit.

A page that renders many independent widgets, each hitting a service. A batch job processing records with per-record API calls. A webhook handler that fans out notifications. An agent making parallel tool calls, which is the newest version and the one where the fan-out width is often decided by a model rather than by a developer, so the only reliable bound is the one you impose.

The question is the same in all of them: what is the maximum number of simultaneous operations this can produce, and is that number knowable from reading the code? If it is a product of two variables, it is unbounded in practice.

The honest limit

A ceiling trades tail latency for reliability, and under sustained overload it does not save you. If demand genuinely exceeds what the provider will serve, bounding concurrency means everything is slow rather than some things failing, and at some point slow enough is indistinguishable from failed.

The ceiling is a tool for handling bursts and for preventing self-inflicted saturation, not for scaling past a real capacity limit. When the queue is permanently long, the answer is more provider capacity, better filtering before the calls, or fewer candidates, not a different number in the semaphore.

FAQ

Why did making my outbound calls concurrent cause failures rather than speed? Because concurrency without a ceiling scales with the product of request rate and fan-out width, neither of which is a constant. Under simultaneous users that exhausts connection pools, trips provider rate limits or saturates the runtime, and the result is refusals rather than the delays you were trying to remove.

How do I choose a concurrency limit? From whichever constraint binds first: connection pool size divided by expected concurrent requests, provider rate limit divided by expected request rate, or what your runtime handles without degrading unrelated work. Start from the binding constraint rather than from a round number, and add a global limit as well, since a per-request ceiling with unlimited requests still reaches the provider's limit.

Why are refusals worse than slow responses? They cluster at peak, so they hit a large fraction of concurrent users rather than being spread thinly. They trigger client retries, which multiply load exactly when the system is saturated. And they report as provider errors, so the investigation goes to the provider, who was fine.

Is an unbounded queue a safe alternative to dropping work? No. If work arrives faster than it drains, the queue grows until requests time out anyway, having consumed resources throughout. Pair the ceiling with an explicit decision about what to do when the queue is long, such as returning the best answer found within the budget.

Does this apply to parallel tool calls from an agent? Especially. The fan-out width is frequently decided by a model rather than by a developer, so nothing in your code states the maximum. The only reliable bound is the one you impose, and it has to be a shared limit rather than a per-invocation one.

When does a concurrency ceiling stop helping? Under sustained overload. If demand genuinely exceeds what the provider will serve, bounding concurrency means everything is slow rather than some things failing, and slow enough becomes indistinguishable from failed. At that point the answer is more capacity or better filtering, not a larger semaphore.