A recurring gift is a donor's standing instruction to give on a schedule, and it multiplies every webhook hazard by the number of times it runs. The payment processor retries delivery for up to three days, can deliver the same event more than once, and does not guarantee ordering. With a single gift those hazards apply once. With a recurring gift they apply on every charge, for as long as the donor keeps giving. A mishandled recurring charge is not a reporting error. It is a donor charged twice and thanked for both, or a donor thanked for a renewal that actually failed, and that is trust that does not come back.
A recurring gift is a stream of events, and the stream arrives late, twice and out of order.
When a donor sets up a regular gift, the organisation does not charge them each time. The payment processor does. It holds a subscription: the donor's saved payment method, an amount and a schedule. When a payment falls due, the processor attempts the charge and tells the organisation what happened by sending events. One renewal can produce several of them: a charge created, a payment attempted, a failure because the card had expired, a later retry that succeeded. Long afterwards a refund or a dispute may arrive about a charge everyone thought was settled. The organisation's entire view of the subscription is assembled from these messages.
The messages are delivered at least once. The processor sends an event and waits for the organisation's server to acknowledge it. If no acknowledgement comes back, because the server was down or slow or the reply was lost on the way, the processor sends the event again, and keeps trying for up to three days. That guarantees nothing is lost, and it also guarantees that some events arrive twice, because a lost acknowledgement looks exactly like a lost message. Ordering is not guaranteed either. Events are sent and retried independently, so the success from a retry can arrive before the failure from the attempt that came before it.
There are two ways to handle a stream like this. The first treats each message as an instruction: on payment succeeded, mark the subscription active and add to the donor's total. That is only correct if every message arrives exactly once and in order. The second treats each message as a fact to be recorded. Every event is appended to a ledger, an event already in the ledger is ignored, and the subscription's state is calculated from everything recorded so far, arranged in the order it actually happened. A late event slots into the history where it belongs. A duplicate changes nothing. Engineers call the result exactly-once in effect: the stream is messy, and the outcome is the same as if it had been clean.
A ledger also answers questions a counter cannot. Why does this donor show as lapsed? Read their history. Was a particular charge refunded? There is an entry for it. A balance column holds only its latest value and forgets how it got there. When it is wrong it cannot be corrected, only overwritten with a different guess.
Recurring giving is an event-ledger problem. Every event the processor sends is appended to a record that is never edited, duplicates are dropped by their event identifier, and the order things happened in is reconstructed from the record instead of being assumed from the order they arrived. The obvious approach, updating a balance as each webhook lands, is only correct if delivery happens exactly once and in order. It happens neither way.
Event handlers fail quietly, because every message looks fine on its own.
A handler that acts on messages as they arrive is easy to write and easy to test. In a test environment each message arrives once, in order, promptly, and every test passes. The bugs exist only in combinations: this message twice, that one before this one, an old message landing after a newer one has already been applied. Those combinations are rare for any one message and routine across the volume a recurring programme produces over time.
When the state goes wrong, nothing records how. The balance is just a number and the subscription status is just a word, each set by whichever message happened to arrive last. Investigating means trawling logs for messages that were each individually correct. Repairing means overwriting the state with a new guess. Meanwhile the donor has been charged twice, or sent a lapsed notice for a gift that went through, or thanked for a gift that bounced.
The reflex, and the fix.
Process each webhook on arrival and update a balance
It is the natural reading of a webhook, and it works perfectly in testing, where delivery is clean.
Duplicates count twice, and a late or out-of-order event applies an old change on top of a newer one. The state is corrupted in a way that no single log line reveals.
Append every event, derive state from the ledger
A ledger and a derivation step to build in place of a column to increment, with reconciliation on top of both.
Exactly-once in effect over a stream that is at-least-once and unordered: duplicates dropped, and ordering reconstructed from the record.
Arrival order carries no information worth trusting. So state is computed from the record of what happened, and nothing changes merely because a message arrived.
Deduplicate on the event identifier
An identifier that has been seen before is dropped without being applied. Over a retry window of up to three days, this is the whole defence against the same event arriving more than once.
An append-only event ledger
Every accepted event is appended and never edited. The history of a subscription can be read back in full, so its current state never has to be inferred from a single stored value.
State derived from the ledger
Subscriptions and contributions are computed from the record. A late event that arrives out of order is placed where it belongs in the history, so it cannot overwrite something newer.
Reconciliation against the processor
The derived state is compared with the processor's own view, because the ledger can only be as right as the events it received.
What each choice cost, and what it bought.
ARecording events, never executing them
Each incoming message is treated as a fact and appended to the ledger. Nothing about the subscription changes because a message arrived. That costs more storage and a derivation step, and reading the current state of a subscription means computing it. In exchange the result is the same whatever order the events arrive in and however many times each one arrives, and the full history is always there to read. Executing messages on arrival is simpler and only correct under exactly-once, ordered delivery, which the processor explicitly does not offer.
BThe processor's event identifier as the deduplication key
Duplicates are recognised by the identifier the processor gives each event and by nothing else. That puts total reliance on that identifier being sound, and it accepts that a replay under a new identifier would pass straight through this check. What it buys is exact deduplication with no guessing. The alternative, matching events by their content, fails badly for recurring gifts in particular. Successive renewals are identical by design, the same donor, the same amount, the same description, and content matching would merge genuinely separate charges into one.
CState that can always be rebuilt
The current state of a subscription is a product of the ledger, so it can be recomputed from the ledger at any time. That means the derivation logic has to be written carefully, since it is now the definition of what the state is. It also means a bug in that logic can be fixed and the corrected state rebuilt from history, with no hand repair of individual records. When state is stored and edited directly, every bug leaves behind damaged records that someone has to find and fix one at a time.
DReconciliation as the backstop for the ledger itself
The derived state is compared with the processor's view on a standing basis. That is another job to run and another report to read. It catches the two things the ledger cannot: events the processor never managed to deliver, and a change replayed under a new identifier. Without it those cases are wrong silently. With it they are wrong for a while, visibly, and then corrected.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
No measured figures are published for this engagement. Outcomes are described qualitatively.
A handler that increments a balance on every webhook, counts every redelivery twice, and silently corrupts a subscription when an event arrives three days late and out of order.
An append-only ledger keyed on the processor's event identifier, with subscription and contribution state derived from it, reconciled against the processor, and arrival order trusted for nothing.
What it owns, and what it hands to a person.
The ledger gives exactly-once in effect over a stream that is at-least-once and unordered, and it rests on the processor's event identifier being a sound key for spotting duplicates. If the same underlying change were ever replayed under a new identifier, deduplication would not catch it. Reconciliation would, later and visibly. Slow and visible is the right trade against fast and wrong.
Any at-least-once event stream, webhooks or queues alike, takes the same treatment. Deduplicate on the event identifier, append to a ledger, derive state from it, and never trust the order things arrived in.
At-least-once delivery means some messages arrive twice, because a lost acknowledgement looks identical to a lost message. Design every handler on that assumption.
Treat incoming events as facts to record, not instructions to execute. Recording is safe under duplicates and reordering, and executing is not.
Deduplicate on the sender's event identifier. Recurring payments are identical by design, so matching on content will merge charges that are genuinely separate.
Never trust arrival order. Reconstruct the order things happened in from the events themselves and check it against the source system.
A ledger can explain any figure it produces. A stored balance cannot, and a number that cannot explain itself cannot be repaired with confidence.