Distributed SystemsField engagement

When adding servers stops helping, the problem is not capacity. It is that everything is waiting on one shared resource, and the work is finding which one.

Fractional CTO and lead engineer.
74%
of request time, in the database
p99
not the mean
no
model anywhere
averagep99the kneeloadlatency
Under load the average looks fine while the tail falls off a cliff. Users feel the slowest requests, not the mean.
What was at stake

A system under load does not degrade gently. Users experience the slowest requests, not the average one, and under contention the tail explodes long before the mean moves, so the dashboards look survivable while a meaningful fraction of customers are having an unusable experience. Then the second-order effect arrives: slow responses trigger client retries, retries add load, load slows everything further, and a slow system becomes a down one. Meanwhile the cloud bill was climbing, because the response to slowness had been to add capacity and the capacity was not helping.

The constraint

A stateless tier scales with the line: add a replica, get proportional throughput. But the moment requests contend for one shared resource, a database primary, a lock, a hot row, every new replica adds contention instead of capacity and throughput turns down rather than up. At that point the autoscaler is actively making things worse while reporting that it is working. The actual question was never how much more capacity. It was which resource everything is waiting on, and that is a measurement, not a guess.

app tier12%
cache8%
waiting on the database74%
network6%
Where the request time actually goes. Scaling the app tier does nothing when the system is waiting on the database.
The fork

The reflex, and the fix.

Road not taken

Add replicas and autoscale the app tier

Pull

One configuration change. Buys time at first, and feels like progress.

Why not

Once the bottleneck is the database, more application replicas pile load onto it and make the cliff steeper while the bill climbs.

Road taken

Find the saturated resource, then fix it

Accepted

Measure before changing anything, fix the real constraint, and accept that the bottleneck will move and the work will be done again.

Bought

Throughput that actually rises, and a system that sheds load instead of cascading.

Decision

Scale the system the load actually has, found by measurement, not the one the dashboard makes easy to scale.

How it was built
01Diagnose
02Stateless tier
03Data tier
04Async + backpressure
05Protect
06Re-measure
re-measure, the bottleneck has moved
01

Measure before you scale anything

Distributed tracing across the request path showed where the time actually went: 74% of request time was in the database, not in the application tier that had been getting the replicas. Once that was on a trace rather than in a meeting, the work was obvious and the spend stopped going to the wrong place.

02

Make the easy part actually easy

A replica holding session or local state cannot be added or moved freely. Pushing state to a shared store makes the application tier disposable and elastic, so it scales with the line as it was supposed to.

03

The data tier is the wall

Reads are the tractable half: index and query work first, then pooling, replicas, and caching, each trading a little staleness for a lot of headroom. Writes are the hard half, because a single consistent primary has a ceiling, and getting past it means partitioning, the most powerful and most expensive move available.

04

Take work off the request path

Only the work a user is waiting on belongs on it. Everything else goes to a bounded queue that absorbs spikes and lets workers drain safely. When the queue fills, the system sheds load deliberately, because a slow response can be retried and a collapse cannot.

How it was measured

A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.

p99 and p999 latencyThroughput at SLOHeadroom to the next kneeLoad shed versus collapse past the limit
figures

What it produces
Without this discipline

A rising cloud bill, a tail that still spikes, and an outage every time traffic jumps, because the contended resource was never the app tier that got scaled.

This system

Throughput that rises with capacity because the real bottleneck was fixed, a tail held under load, graceful shedding instead of cascades, and a known answer to where the next wall is.

diagnose first74% found by tracingthe data tier is the walldegrade gracefully
The operating envelope

What it owns, and what it hands to a person.

Handled with confidence
Load within modeled capacity
The diagnosed bottleneck addressed
Tail within SLO
Flagged for review
Approaching the next knee
Behavior past the modeled limit
Out of scope by design
The ceiling of a single consistent write path
Premature sharding
The honest limit

Scaling is iterative. The bottleneck always moves and there is no final state, only the next constraint. Some limits are fundamental: a strongly consistent write path does not scale forever, and partitioning buys throughput at the cost of real and permanent complexity, which makes it a last resort rather than a first.

What it generalizes to

There is nothing in this program that an AI vendor could have sold and nothing a model could have solved. It is measurement, contention, and the architecture of the data tier. It also shows the Diagnose stage doing the thing it is actually for: preventing an expensive wrong action. The reflex here, adding capacity, was already underway, already costing money, and already making the failure worse.

How we engage

You have a system like this one.
Tell us where it stands.

Whether it is failing, not yet built, or about to meet a scale it has never seen, we can tell you what we see.

Start a conversation
mostafa@opulion.dev · Response within 24 hours · By inquiry