Sync The Commercial Record From Day One, And Make The Sync Idempotent

Mostafa Dhouib·10 min read

Sync The Commercial Record From Day One, And Make The Sync Idempotent

The short answer. If your platform handles the transaction and a separate commercial system holds the customer relationship, build the sync with the first transaction rather than later. Retro-fitting it means back-filling history into a store that people have already been editing by hand, reconciling two sets of records that were never reconciled, and unpicking duplicates created during the gap. And the sync has to be idempotent on day one, because a retried webhook that creates a second contact or a second deal is not a technical curiosity. Someone gets contacted twice, a pipeline figure double counts, and a commission is paid on a deal that does not exist.

A platform transacts. Somewhere else, usually a hosted commercial system, sits the record of the customer relationship: who they are, what has been discussed, what is in the pipeline, who owns the account.

Both of these stores describe the same companies and the same money. Neither was designed with the other in mind.

The decision that gets deferred is how they stay in agreement, and deferring it is more expensive than almost anything else on the list.

Why retro-fitting costs so much more

The gap between launch and integration is not idle time. It is time in which divergence accumulates.

While there is no sync, the commercial system is maintained by people. They type company names in whatever form they have to hand. They create a contact for someone who already exists under a different address. They move a deal to closed because they heard it closed. Every one of those is a legitimate action by a person doing their job, and every one is a record that no longer matches anything on the platform.

So the retro-fit is not an integration project. It is a data reconciliation project with an integration at the end of it. You have to match platform parties to commercial records with no shared key, decide what happens to the ones that match nothing, back-fill transaction history into deals that were created by hand with different stages, and resolve every case where the two stores disagree about a figure that someone has already reported.

Worse, you have to do it while both sides keep changing, because you cannot freeze a business to run a migration.

Built with the first transaction, none of this exists. The first record created on the platform creates its counterpart. The shared key is established at creation, by the system, which is the only moment it is free. People in the commercial system work with records that already carry that key, so their edits land on the right object.

The asymmetry is stark. On day one it is a few hundred lines and a decision about ownership. A year in it is a project with a data cleanup inside it and a long tail of disputed figures.

Webhook-driven, with a durable queue behind it

The mechanism that works is events out of the platform, consumed by a worker that applies them to the commercial system.

Emit from the transition, not from the handler. When a transaction moves state, that transition is already being recorded as an entry. The sync should be driven from those entries rather than from the request handler that caused them. Driving it from the handler means a transition applied by a background job, a correction or an administrative action does not emit, and the sync silently misses exactly the events that matter most.

Put a durable queue between the two. Never call the external system inline in a request. It will be slow, it will rate limit you, and it will be down. A request that fails because an unrelated system is unavailable is a transaction you lost for no reason. Write the event, acknowledge the request, let a worker deliver it with retries and backoff.

Expect at-least-once, both directions. Your queue will redeliver. The external system's webhooks, if you consume them, will also redeliver, and they will arrive out of order. Nothing in this path is exactly once, which is the whole reason the next section exists.

Keep a per-record sync state. For each platform object, record what was last successfully pushed and when. That is how you detect a record that has stopped syncing, which is otherwise invisible, because a stalled sync produces no error on the platform and no missing data that anyone notices until a report is wrong.

Alarm on silence. A sync that delivers nothing for a day looks exactly like a quiet day. If the usual cadence is known, a worker that has emitted nothing by the expected time should raise an alarm on absence rather than on an error.

Idempotency, because duplicates are commercial damage

This is the part that must be right from the first delivery, and it is the part that feels like over-engineering until the day it does not.

A duplicate in a commercial system is not a tidy-up task. A duplicate contact means someone is emailed twice by two owners, which the customer sees. A duplicate deal means the pipeline total is overstated, which means a forecast is wrong, which means decisions are made on it. In organisations where commission is calculated from closed deals, a duplicate is money paid out against nothing.

So the integration needs a stable external identity, not just a retry policy.

Use an upsert keyed on your identifier. Every platform object carries its own identifier. Write that identifier into a dedicated field on the commercial record and make every write an upsert matched on that field. Then a redelivered event finds the existing record and updates it rather than creating a sibling.

Never match on a mutable human attribute. Matching on company name, email address or deal title will fail on both sides. Different spellings create duplicates, and shared or changed addresses merge two parties who should be separate. These attributes change, and the key must not.

Make the key the first thing written. If the create and the key-setting are two calls, a failure between them leaves a record with no key, and the next delivery creates another one. The identifier belongs in the create payload.

Record the association on your side too. Store the external record identifier against your object once you have it. That gives you a direct update path, lets you detect the case where two external records claim the same platform object, and makes the association auditable.

Decide what a replay of an old event should do. Out-of-order delivery means a stale event can arrive after a newer one. Carry the transition's position or timestamp and refuse to apply an update older than what the record already reflects. Without that, a retry from an hour ago can quietly revert a correction.

Test the duplicate case explicitly. Deliver the same event twice, deliver two events out of order, and assert that exactly one record exists in the expected state. This is a cheap test that catches the expensive failure, and it is almost never written.

Which system owns which fact

Most sync pain is not transport. It is two systems that both believe they are authoritative about the same field.

The rule is to assign ownership per fact, write it down, and let the non-owner hold a copy that is read-only in practice.

The platform owns everything derived from a transaction. Amounts, volumes, dates, state, settlement status. These are computed from the entry log, and a person editing them in the commercial system is creating a figure with no provenance. If the commercial system permits such edits, the sync should overwrite them on the next push, and everyone should know that in advance.

The commercial system owns the relationship. Account ownership, pipeline stage before a transaction exists, notes, next actions, marketing consent. The platform should not invent these, and if it needs them it should read them rather than maintain its own version.

Identity is shared, with one creator. Exactly one side creates a party record, and the other follows. Whichever you choose, both-create is the design that produces duplicates, because two systems independently deciding a company is new will each make one.

Name the overlaps explicitly. There are always a handful of fields both sides want to write, usually contact details and status. For each one, pick an owner. An unowned field is a field that will be inconsistent, and the inconsistency will be discovered in a customer conversation.

A useful discipline: for any number a person might quote externally, there should be exactly one system you would check to verify it. If the answer is "either", the ownership question has not been answered.

The honest limit

A day-one sync is cheaper than a retro-fit, and it is not free. You are taking a dependency on an external system's data model early, when your own is still moving, and some of that mapping will be rewritten. That is a real cost and it is smaller than the alternative.

Idempotency on the push prevents duplicates you create. It does nothing about duplicates created by people typing into the commercial system directly, which will continue. Deduplication there is an ongoing operational practice, not something the integration solves.

The approach also assumes the external system lets you store your identifier on the record and match on it. Most do, through a custom field. Where that is not available, or where the matching is limited to a handful of built-in fields, the design gets considerably harder and may need a mapping table on your side as the only durable association.

And ownership written down is not ownership enforced. If the commercial system allows a transaction amount to be edited and people edit it, the next push overwrites their work, which is correct and will still be experienced as the integration losing data. That conversation is worth having before the first overwrite rather than after it.

FAQ

Why is retro-fitting the sync so much more expensive than building it early? Because the gap fills with divergence. People maintain the commercial system by hand, creating records with no shared key, inconsistent names and stages that reflect hearsay. The retro-fit becomes a reconciliation project: match parties with no common identifier, back-fill history into hand-made deals, and resolve every figure the two stores already disagree about, all while both keep changing.

Why does the integration need idempotency from the first delivery? Because queues and webhooks are at-least-once, so the same event will arrive twice, and a duplicate in a commercial system is commercial damage. A duplicate contact means a customer is approached twice by two owners. A duplicate deal overstates the pipeline, which corrupts a forecast, and where commission is calculated from closed deals it pays out against nothing.

What should the integration key on? Your own platform identifier, written into a dedicated field on the external record, with every write performed as an upsert matched on that field. Never match on company name, email address or deal title. Those change, and they produce duplicates when spellings differ and false merges when an address is shared between two parties.

Should the external call happen inside the request? No. Write the event, acknowledge the request, and let a worker deliver it with retries and backoff. An inline call makes your transaction path depend on a system you do not control, so a rate limit or an outage there becomes a lost transaction here. The queue also gives you the retry behaviour that idempotency is there to make safe.

How do I decide which system owns a field? Assign ownership per fact and write it down. The platform owns anything derived from a transaction, because those figures come from the entry log and a hand-edited version has no provenance. The commercial system owns the relationship: account ownership, notes, next actions, consent. For any number someone might quote externally, exactly one system should be the one you check.

How do I know the sync has stopped working? Keep per-record sync state and alarm on absence. A stalled sync produces no error on the platform and no visibly missing data, so it looks like a quiet period until a report is wrong weeks later. If the normal cadence is known, a worker that has delivered nothing by the expected time should page, rather than waiting for a failure to fire.