Two Postgres Queues, Opposite Failures, Same Drill: Know Which Way Yours Breaks

Mostafa Dhouib·7 min read

Two Postgres Queues, Opposite Failures, Same Drill: Know Which Way Yours Breaks

The short answer. Run one drill against your job queue: start a long job, kill the worker, and watch what happens. The two most widely used database-backed queues answer that drill in opposite directions. One treats a job still active past a fixed interval as abandoned and re-runs it, whether the worker died or is simply slow, so a legitimately long task gets executed twice. The other holds a dead worker's job for hours before anything else can take it, so a crashed run stalls. Both defaults are defensible. Neither is safe to inherit without knowing which one you have, because the failure you get is the opposite of the failure you prepared for.

Queue selection usually happens for reasons that have nothing to do with failure behaviour. It is already in the stack, it has a nice API, somebody used it before. Then the queue runs correctly for months, because the happy path in a job queue is genuinely easy, and the first time anyone learns its failure semantics is during an incident.

There is one drill that surfaces almost all of it, and it takes ten minutes.

The drill

Enqueue a job that takes longer than your queue's visibility or timeout setting. Start a worker. Kill it mid-job, not gracefully. Watch what the queue does.

There are only a few possible outcomes and each tells you something you need to know.

The job is picked up again immediately by another worker. The job is picked up after a delay. The job is never picked up again. Or the job is picked up again while the original worker is still running it, which is the interesting one.

Opposite answers from two popular queues

The two most common Postgres-backed job queues differ here in a way that is worth stating plainly, because the names of the settings do not reveal it.

One of them, on default settings, treats any job still marked active past a fixed interval as abandoned, and retries it. The interval is in the region of fifteen minutes. Crucially, this is based on elapsed time rather than on worker liveness. A worker that is alive, healthy and halfway through a legitimately long task gets its job re-run by somebody else while it is still working on it.

The other holds a dead worker's job for substantially longer, up to several hours, before another worker can claim it. That avoids the duplicate-execution problem entirely, and it means a worker that genuinely crashed leaves its job stuck for a long time.

So on the same drill, one queue duplicates and the other stalls.

Neither is a bug. The first optimises for progress, assuming jobs are short and duplication is survivable. The second optimises for safety, assuming duplication is worse than delay. Both assumptions are correct for the workloads their defaults were chosen against, and at least one of them is probably wrong for yours.

Why this bites hardest on long-running work

The duplicate-execution direction is the dangerous one, and it has become much more common recently because of what people are putting into queues.

A job that takes thirty seconds is never going to trip a fifteen-minute threshold. A job that wraps a model-driven process, a build, a large import or anything with variable duration regularly will. The queue is not distinguishing "this worker is dead" from "this work is slow", because at the database level those look identical: a row marked active with an old timestamp.

The result is two workers doing the same expensive thing concurrently. If the work has side effects, and queued work usually does, you now have the side effect twice, from a queue you believed was delivering each job once.

Worse, the duplication is correlated with slowness, so it appears under load and disappears when you go looking for it.

What to do about it

Run the drill and write down the answer. Not what the documentation implies, what your configuration does. Ten minutes, once, and it becomes a fact rather than an assumption.

Make the work idempotent regardless. Every one of these queues is at-least-once. That is not a defect, it is the only thing a queue can honestly offer across a crash boundary, because the alternative requires the queue and your side effect to commit together. Give each job an identity, make the handler safe to run twice, and the queue's exact behaviour stops being load-bearing.

Heartbeat, if the queue supports it. The ambiguity is entirely about not knowing whether the worker is alive. A worker that reports in converts an inference from elapsed time into a fact, and lets you set a short timeout without punishing legitimately long work.

Set the timeout from your actual duration distribution. Not the median. The tail. If your ninety-ninth-percentile job takes eleven minutes, a fifteen-minute threshold is not the comfortable margin it appears to be.

The same drill applies further out

This is one instance of a general class, and the class is worth naming because it recurs across everything that supervises other work.

Cancellation has the same shape. In at least one widely used workflow engine, cancellation is only delivered to work that heartbeats. A wrapped process that never reports in cannot be cancelled, no matter what the interface offers, so a cancel button that appears to work and silently does nothing is the default outcome rather than an edge case.

Budget and cost caps have the same shape. A cap that resets when a task is resumed or continued stops being a control at exactly the moment a task is going back for more work, which is when cost accumulates.

In each case the component is behaving correctly and the operator has inherited a behaviour they never chose. The remedy is identical: find out what it does under failure, rather than under success.

The honest limit

The specific intervals quoted here are defaults, and defaults change between versions. The numbers matter less than the direction, and the direction is what you should verify yourself, on your version, with your configuration.

The drill also only covers one failure: an abrupt worker loss. It says nothing about what happens when the database connection drops mid-commit, when the job payload cannot be deserialised after a deploy, or when a worker is running old code. Those are separate drills, and the habit of running them is the point rather than this particular one.

FAQ

What happens if a worker dies while processing a job? It depends entirely on which queue you are using, and the two most common Postgres-backed ones answer in opposite ways. One re-runs the job after a fixed interval based on elapsed time rather than worker liveness, so it duplicates. The other holds the job for hours before anything else can claim it, so it stalls.

Why would a queue re-run a job whose worker is still alive? Because at the database level a healthy worker on a long task and a dead worker look identical: a row marked active with an old timestamp. A queue that infers abandonment from elapsed time cannot tell them apart, so a legitimately slow job gets picked up by a second worker while the first is still running it.

Is my job queue exactly-once? No. Every database-backed queue is at-least-once across a crash boundary, because delivering exactly once would require the queue and your side effect to commit together. Give each job an identity and make the handler safe to run twice, and the queue's precise behaviour stops being load-bearing.

How should I set my job timeout? From the tail of your actual duration distribution rather than the median. If your ninety-ninth-percentile job runs for eleven minutes, a fifteen-minute threshold is not the margin it looks like, especially for work whose duration varies with input.

Why can't I cancel a running agent or wrapped process? In at least one common workflow engine, cancellation is only delivered to work that heartbeats. A wrapped process that never reports in cannot receive it, so the cancel control appears to function and does nothing. That is the default behaviour rather than an edge case.

What is the fastest way to learn my queue's failure behaviour? Enqueue a job longer than the timeout, start a worker, kill it abruptly, and watch. The outcome tells you whether you get duplication, a stall, or correct recovery, and it takes about ten minutes. Write the answer down, because it is configuration-specific and changes between versions.