Do Injected Faults Track Real Bugs? The Case for Mutation Testing

Mostafa Dhouib·August 25, 2026·6 min read

Do Injected Faults Track Real Bugs? The Case for Mutation Testing

The short answer. Across 357 real faults and 230,000 mutants in five applications, a coupling relationship held for 73 percent of real faults, and mutant detection tracked real fault detection more strongly than statement coverage did. It is the evidence that "make your test catch an injected fault" is a valid proxy for catching real ones.

The whole negative control argument rests on a substitution: you cannot inject the bugs you do not know about, so you inject small artificial ones instead and treat catching those as evidence. That substitution needs justifying, and it has been.

The study

Just, Jalali, Inozemtseva, Ernst, Holmes and Fraser, at FSE 2014.

They isolated 357 real faults from five real applications, narrowed from 1,179 candidate consecutive revision pairs, keeping only faults that could be cleanly isolated, meaning the fix changes the faulty behaviour and nothing else. The applications were Chart, Closure, Commons Math, Joda-Time and Commons Lang, totalling 321,000 lines of code. This fault set went on to become the basis of the widely used Defects4J benchmark.

Against those they generated 230,000 mutants using the Major mutation framework, with operators that replace constants, replace arithmetic, relational, logical and conditional operators, modify branch conditions, and delete statements.

Both developer written and automatically generated suites were evaluated, the latter via EvoSuite, Randoop and JCrasher. Critically, the analysis controlled for statement coverage, so the results are not simply reporting that more code was executed.

The coupling result

A coupling relationship exists for 73 percent of the real faults. That means a test that detects the real fault also kills at least one mutant, so mutation adequate testing would have forced detection of those faults.

Two further numbers make the result usable rather than merely reassuring.

Controlling for coverage, on average only about 2 mutants couple to a single real fault, and they are disproportionately conditional operator replacement, relational operator replacement and statement deletion mutants. A small, specific set of operators does most of the work, which is what makes cheap mutation testing possible at all.

And the sharpest one. Of 480 triggering test pairs analysed, adding the fault detecting test raised mutation score in 75 percent of cases but raised statement coverage in only 46 percent. The test that catches the bug frequently adds no coverage at all while killing mutants.

That single comparison is the clearest available statement of what mutation score sees and coverage does not. If your metric does not move when somebody adds the test that catches a real bug, your metric is not measuring test quality.

The correlation result

Mutant detection is positively correlated with real fault detection, and more strongly than statement coverage is.

The Wilcoxon signed rank test was significant for all subject programs except Joda-Time, and the Vargha-Delaney A12 effect sizes were at or above 0.71, which is conventionally "large", for most programs.

Joda-Time being the exception is worth noticing rather than skipping. It is also the program where Inozemtseva and Holmes found the coverage to effectiveness correlation fell essentially to zero once size was controlled. Something about that codebase resists both measures, which is a reminder that these are population level results and your codebase is one sample.

The limits, stated plainly

Mutation testing is strong. It is not omniscient, and overselling it here would be the exact failure this pillar is about.

27 percent of the real faults were not coupled to any generated mutant.

Of those, 17 percent of all real faults are not coupled to any mutant even in principle under commonly used operators.

The uncoupled faults are dominated by a specific and recognisable set:

Algorithm rewrites and large logic changes, 37 cases. A mutant is by construction a small change, so it cannot represent a fault that is a wholesale re-implementation of the wrong approach.

Code deletion faults, where the bug is that something is missing.

Numerical, NaN and floating point handling faults.

Context sensitive faults that depend on broader program state.

The authors note mutation testing improves with stronger operators, and that it has a genuine ceiling. It is the best available proxy, not truth, and a suite that kills every mutant has demonstrated something real and bounded.

The prior disagreement, and why this is the citation of record

This question had been asked before and answered both ways.

Andrews, Briand and Labiche, ICSE 2005, found mutants to be a good proxy for real faults.

Namin and Kakarla, ISSTA 2011, replicated on the same subject program with a different tool and found the correlation weak.

Just et al.'s contribution over both is scale, 357 real faults and 230,000 mutants across five applications, and coverage control. That is why it is the study to cite, and why the earlier disagreement is settled rather than ignored.

What this licenses you to do

It licenses the practice, not a number.

Inject a fault into the code a test protects, and confirm the suite kills it. A survivor is a located test gap, and the evidence says that closing it is a reasonable proxy for catching real faults, three quarters of the time, on that population.

It does not license chasing a mutation score, for reasons that are separate and equally well measured: equivalent mutants make one hundred percent unreachable, and their prevalence runs 4 to 39 percent. That is covered in stop chasing 100 percent mutation score.

The general argument is in a test that has never failed is a decoration, and the smallest possible version of this practice, requiring no tooling at all, is a single manual mutation on the module you would least like to be wrong about.

FAQ

Does mutation testing find real bugs? It tracks them. Coupling holds for 73 percent of 357 real faults, and mutant detection correlates with real fault detection more strongly than statement coverage does, with large effect sizes for most programs studied.

Is mutation score a valid quality measure? It is the best available proxy rather than truth, and it has a measured ceiling. 17 percent of real faults couple to no mutant even in principle under common operators.

Which mutation operators matter most? Conditional operator replacement, relational operator replacement and statement deletion do most of the coupling work with real faults, which is why cheap mutation testing is possible.

What kinds of bug does mutation testing miss? Algorithm rewrites and large logic changes, which a small mutation cannot represent, plus code deletion faults, numerical and NaN handling, and context sensitive faults depending on broader program state.

Does mutation testing just measure coverage again? No. The analysis controlled for statement coverage, and of 480 triggering test pairs, adding the fault detecting test raised mutation score in 75 percent of cases and statement coverage in only 46 percent.