What a Codec Does to a Watermark Is a Ceiling, Not a Bug

Mostafa Dhouib·10 min read

What a Codec Does to a Watermark Is a Ceiling, Not a Bug

The short answer. Published third-party evaluations of audio watermarking report detection performance near 0.97 on clean audio collapsing toward chance, around 0.57, once the audio has passed through certain neural codecs. That is not a detector that needs more training. A neural codec re-synthesises audio to be perceptually similar while discarding almost everything else, and a watermark is by design an imperceptible perturbation, which puts it in exactly the category the codec throws away. The result is a ceiling set by the transform. Recognising a ceiling as a ceiling is what stops a team spending a quarter improving a component that is already at its limit, and it changes the design question from how to detect better to what to do when detection is not available.

There is a specific failure of engineering reasoning that this example illustrates better than anything else I know.

A number is bad. The component that produced the number is identified as the problem. Work begins on that component. The number does not improve, because the number was never a property of that component in the first place.

What a neural codec actually does to the audio

A conventional codec removes information a model of hearing says you will not notice, and what comes out the other side is still, structurally, the waveform that went in with parts removed. A watermark embedded with some margin usually survives.

A neural codec works differently. It encodes audio into a compact learned representation and then generates audio from that representation. The output is not the input with parts removed. It is new audio, synthesised to be perceptually close to the original, by a system trained to optimise perceptual similarity and nothing else.

Now consider what a watermark is. It is a deliberately imperceptible modification of the signal. Imperceptibility is its defining requirement, because a watermark you can hear is not acceptable in any real product.

Put those two facts next to each other and the outcome is not surprising. The codec is trained to preserve what is perceptible and is under no obligation whatsoever to preserve what is not. The watermark has announced itself, by construction, as part of the category that can be discarded. It is not that the codec attacks the watermark. It is that the codec has no reason to carry it, and a learned bottleneck keeps only what its objective rewarded.

Hence the collapse toward chance. At around 0.57, a detector is barely doing better than guessing, which means the information it needs is substantially absent from what it is being shown. No decision rule recovers information that is not present in the input.

Ceiling or bug: the distinction and the test

A bug is a gap between what the system does and what it could do with the information available. A ceiling is a limit on the information available. Both show up as a disappointing number, and they call for opposite responses.

The practical test is whether the information survived the transform at all.

Take the material after the transform and ask whether any decision rule separates the watermarked items from the unwatermarked ones. Not your current rule. Any rule. Look at the score distributions for both populations and see whether they overlap completely or whether there is separation your threshold is failing to exploit.

If there is separation and your detector is not using it, that is a bug, and it is fixable in the detector. Recalibrate, retrain on transformed material, change the decision rule.

If the distributions are on top of each other, the information is gone. That is a ceiling, and no work on the detector moves it. The honest statement is that detection is not available under that condition.

The second test, which is worth running because it is cheap and often decisive, is to attempt recovery with full knowledge. Give yourself the original audio, the watermark key, and the exact embedding parameters, then try to recover the mark from the transformed audio. This is a strictly easier problem than production detection, which has none of that information. If you cannot recover the mark under those conditions, production detection is not going to, and you have your answer without building anything.

Why the distinction decides what you build next

This is the part that matters commercially, and it is why I keep returning to this example.

If the answer is a bug, the roadmap is a detector roadmap. Train on codec-processed material, raise the embedding strength until it survives, redesign the embedding to place energy where the codec preserves it, and measure the improvement. Those are all reasonable engineering projects with a plausible return.

If the answer is a ceiling, every one of those projects has a return near zero, and the roadmap is a different roadmap entirely.

Detect the condition instead of the mark. You often cannot find the watermark after a neural codec, but you can frequently tell that audio has been through one. The artefacts of re-synthesis are their own signal. A system that reports "no watermark found, and this audio has been re-synthesised" is far more useful than one that reports "no watermark found", because the two statements support completely different decisions.

Move the evidence out of the signal. If the signal cannot carry the mark through the chain, carry provenance alongside the content instead. Signed metadata, a manifest, a registry entry, a perceptual hash lookup. These have their own weaknesses, principally that they are easy to strip, but they fail in a different and more visible way than a watermark that silently does not survive.

Change where in the chain you detect. If detection must be reliable, do it before the transform. That is an architectural decision about pipeline placement rather than a detector improvement, and it is frequently available once someone asks the question.

State the limit in the product. A system that claims detection and quietly cannot deliver it under common conditions is worse than one that scopes its claim, because downstream users build policy on the claim. Writing the gap down is an engineering deliverable.

Notice that none of these is a better detector. That is the point. Mistaking a ceiling for a bug does not just waste the effort spent on the detector. It delays the work that would actually have helped.

Robustness targets against named conditions

The deeper lesson is about how robustness gets specified, and it applies well beyond watermarking.

A single robustness number is close to meaningless, because it is an average over a distribution of conditions that somebody chose and nobody wrote down. Change the mix of conditions and the number changes, with no change to the system. Two vendors quoting robustness figures are frequently not comparing anything.

The alternative is a condition matrix. Name each transform, state the target under it, and measure against each separately.

Enumerate conditions from the deployment, not from a benchmark. The chain your content actually passes through. Which codecs, at which bitrates, which platform re-encodes, which enhancement stages. A condition list copied from a paper measures your system against someone else's deployment.

Set a separate target per condition, and allow some to be zero. A target of zero under a condition that destroys the signal is an honest specification. A blended target that hides it is not. Writing a zero down forces the design conversation about what the system does under that condition, which is the conversation that produces the actual fix.

Separate the detection rate from the false positive rate at each condition. A detector tuned to survive a harsh transform usually becomes more willing to call a mark where there is none, and in provenance work a false accusation is often the more expensive error. A single robustness figure conceals this trade completely.

Keep the clean baseline in the table. The near-0.97 figure matters precisely because it sits next to the 0.57 one. Together they say the detector works and the condition is the problem. Either alone says much less.

The honest limit

The 0.97 and 0.57 figures are published third-party results on particular watermarking schemes against particular neural codecs on particular audio. They are not my measurements, they are not universal, and they are not a statement about every scheme or every codec. The field moves, and some schemes are deliberately designed to place their energy where learned codecs preserve it. Treat the pair as a demonstration that the failure mode exists and is severe, not as a constant.

The reasoning also has a boundary that is easy to overstate. Not every transform is a ceiling. Conventional lossy compression, resampling and modest noise are usually survivable with adequate embedding strength, and a detector that fails under those probably does have a bug. The ceiling argument applies specifically where a transform re-synthesises the content rather than degrading it, and that distinction has to be established per transform rather than assumed.

And a ceiling can be moved, just not from inside the detector. Stronger embedding, placement in features the codec must preserve, or embedding at a different stage of the pipeline all change the problem. What cannot change it is more detector capacity applied to material from which the information has already been removed.

FAQ

Why does a neural codec destroy an audio watermark when ordinary compression does not? Because a neural codec re-synthesises the audio from a learned representation rather than removing parts of the original waveform, and it is trained to preserve only what is perceptible. A watermark is by definition an imperceptible perturbation, which places it exactly in the category the codec has no reason to carry. Published results report detection falling from near 0.97 on clean audio to about 0.57, close to chance, under certain neural codecs.

How do I tell whether a bad detection figure is a ceiling or a bug? Compare the score distributions of watermarked and unwatermarked material after the transform. If they separate at all, some decision rule can exploit it and you have a fixable detector problem. If they overlap completely, the information is gone and no detector change recovers it. A cheap confirmation is to attempt recovery with the original audio, the key and the embedding parameters all in hand, since production detection has none of those advantages.

What should I build if the transform really does destroy the watermark? Not a better detector. Detect the condition instead, since re-synthesis leaves its own artefacts and reporting that audio has been re-synthesised supports a real decision. Move provenance out of the signal into signed metadata or a registry, detect earlier in the chain before the transform, and state the gap in the product rather than letting downstream users build policy on a claim that does not hold.

Why is a single robustness number not good enough? Because it averages over a set of conditions that somebody chose and usually did not publish, so changing the mix changes the number without changing the system. A condition matrix with a named transform per row, a separate target each, and detection and false positive rates reported separately is a specification. One blended figure is a marketing artefact.

Is it acceptable to set a robustness target of zero under some condition? Yes, and it is more useful than hiding it in an average. A written zero forces the design conversation about what the system does when detection is unavailable, which is where the real fix usually comes from. A blended target that conceals the same gap produces a product claim nobody can rely on.

Does this mean audio watermarking is not worth doing? No. It means watermarking is reliable against the conditions it was designed for and not against transforms that re-synthesise content, and the specification has to say which is which. Conventional compression, resampling and modest noise are typically survivable with adequate embedding strength. The failure described here is specific to re-synthesis, and it has to be established per transform rather than assumed everywhere.