At Least Once With No Replay

Mostafa Dhouib·8 min read

At Least Once With No Replay

The short answer. Inbound webhooks from the major messaging and social platforms are at-least-once. They retry on failure over an extended window, which means your endpoint will receive the same event more than once. They also typically offer no way to request message history after the fact, so there is no replay, no backfill and no catch-up call. Those two properties together are the problem. One of them guarantees duplicates and the other guarantees that anything you fail to accept during a long outage is gone permanently. The response is to deduplicate on a stable platform identifier and record every receipt from the first day, because neither can be retrofitted over a gap.

A messaging integration looks like the easy part of a platform. There is an endpoint, the platform posts to it, you store the message.

It stays easy until the first incident, and then it turns out the system has two independent data integrity problems that were both present from the beginning and invisible while nothing failed.

The two properties, stated precisely

Platforms that push events to you make a delivery guarantee, and the guarantee is at-least-once.

That means the platform will keep trying until your endpoint acknowledges. If your handler is slow and the platform's request times out, it retries. If your handler returns an error after it has already written the record, it retries. If a network path drops the acknowledgement rather than the request, it retries. The retry schedule typically backs off and continues over an extended window measured in hours rather than seconds.

Separately, and this is the part people do not plan for, these platforms generally provide no API to ask for what you missed. There is no endpoint that returns the conversation history for a window you were down. Whatever was delivered while you were unable to accept it, within the retry window, arrives late. Whatever falls outside the retry window never arrives at all.

Each property alone is manageable. Together they define the shape of the engineering.

Duplicates are not an edge case

Because delivery is at-least-once, the question is not whether you will receive duplicates. It is what your system does when you do.

Without deduplication, a retry creates a second record of the same message. The consequences depend on what the record drives.

If the message populates a conversation view, the user sees the same inbound message twice and loses confidence in the system. If it triggers an automated reply, the contact receives two replies. If it increments a counter, the counter is wrong, and counters derived from message volume feed reporting that someone makes decisions with. If it creates a task or a lead, someone works the same item twice, which is the version that costs money and gets noticed.

The fix is not clever and it must be early. Every platform event carries an identifier that is stable across retries. Store that identifier with a uniqueness constraint, and make acceptance of an event conditional on the insert succeeding. The database enforces the invariant rather than the application remembering to check, which matters because the application will be rewritten and the constraint will not.

Two details do the actual work.

Use the platform's identifier, not your own. A hash of the payload fails when the platform includes a delivery attempt counter or a changing timestamp, because the retry then hashes differently and the duplicate passes. The identifier the platform assigns to the message is the only field guaranteed to be identical across attempts.

Acknowledge before you process. Accept the event, record it, return success, and do the downstream work asynchronously. A handler that does the full processing inline will eventually exceed the platform's timeout, which turns a slow system into a duplicating system.

The receipt log is the other half

Deduplication protects you from retries. It does nothing for gaps, and gaps are the worse failure because they are silent.

What you need is a record of every event received, written before any interpretation, keyed by the platform identifier, with the raw payload and the arrival time. Not the parsed record. The payload as it came.

That log does three things no amount of careful processing can do.

It makes gaps visible. Where platform identifiers are sequential or carry a timestamp, a receipt log lets you reason about what is missing. Without it, a missing message is indistinguishable from a message that was never sent, and you will be arguing with a customer about which it was.

It makes reprocessing possible. When a parsing bug corrupts a field for three weeks, the raw payloads let you rebuild. Without them, the corrupted interpretation is the only version you have, and the platform will not give you the originals back.

It separates receipt failures from processing failures. An event that arrived and failed downstream is a bug you can fix and retry locally. An event that never arrived is a different problem with a different owner. Without a receipt log those two look identical from the inside.

An outage longer than the retry window is permanent

This is the sentence that should appear in the architecture document.

If your endpoint is unavailable for longer than the platform's retry window, the events delivered during that period are lost, and there is no mechanism anywhere to recover them. Not a support ticket. Not a paid tier. The platform retried, you did not accept, and it stopped.

That reframes what availability means for this one endpoint. Most components in a platform can be down for an hour and then catch up, because the work is still queued somewhere. The webhook receiver cannot, because the queue belongs to someone else and it drains on their schedule.

The practical consequences are specific.

Make the receiver the simplest thing in the system. Validate the signature, write the row, return success. No business logic, no joins, no calls to other services. Every dependency you add to that handler is a new way to lose data permanently.

Deploy it so it does not go down with everything else. If the receiver shares a process, a database connection pool or a deployment unit with the main application, then every application outage becomes a data loss event. Separating them is cheap and it is the single highest value decision in this part of the design.

Alert on absence, not just on errors. A receiver that has accepted nothing for an unusual period is an incident even if nothing is throwing. Error-based monitoring is blind to the case where requests stop arriving, which is exactly the case that produces permanent loss.

The honest limit

Deduplication on a stable identifier handles retries of the same event. It does not handle a user genuinely sending the same text twice, which is a legitimate pair of distinct events with distinct identifiers, and no amount of payload comparison will separate those two cases correctly in every instance.

The receipt log makes gaps visible only to the extent the platform's identifiers support inference. Where identifiers are opaque and non-sequential, the log proves what you received and cannot prove what you did not.

And none of this recovers a gap. Every mechanism described here is about bounding the damage and knowing its extent, which is genuinely valuable and is not the same as recovery. If the retry window passes with your endpoint down, the correct action is to tell the affected users that a period of messages was not received, which is only possible if you know when the period started and ended. That knowledge is the real deliverable.

FAQ

What does at-least-once delivery mean for an inbound webhook? It means the platform keeps retrying until your endpoint acknowledges, so your handler will receive the same event more than once. Retries happen when your response times out, when you error after already writing the record, or when the acknowledgement is lost in transit. The retry schedule typically backs off over a window measured in hours.

How should duplicate webhook events be deduplicated? On the stable identifier the platform assigns to the event, stored with a uniqueness constraint so the database enforces the invariant. Do not hash the payload, because platforms often include an attempt counter or changing timestamp that makes a retry hash differently. Acknowledge the event as soon as it is recorded and do downstream processing asynchronously.

Why can a webhook outage cause permanent data loss? Because these platforms typically provide no API to request message history after the fact. There is no replay, backfill or catch-up call. The platform retries for its window and then stops, so any event delivered while your endpoint was unavailable for longer than that window is gone, and no support request recovers it.

What should a webhook receiver actually do? As little as possible. Validate the signature, write the raw payload and its platform identifier to a receipt log, and return success. Every dependency added to that handler, whether a service call, a join or business logic, becomes another way to fail acceptance and lose data that cannot be recovered.

Why keep a raw receipt log rather than just the parsed records? Because the raw payloads are the only copy you will ever have. They let you rebuild after a parsing bug that corrupted a field for weeks, they let you distinguish an event that arrived and failed downstream from one that never arrived at all, and they let you establish the boundaries of a gap. The platform will not return the originals.

What monitoring does this need? Alerting on absence as well as on errors. A receiver that has accepted nothing for an unusually long period is an incident even when nothing is throwing exceptions, and error-based monitoring is structurally blind to requests that stop arriving. That silent case is precisely the one that produces unrecoverable loss.