High Coverage Does Not Mean Safe Code: What the Research Actually Says

Mostafa Dhouib·7 min read

High Coverage Does Not Mean Safe Code: What the Research Actually Says

The short answer. Once suite size is controlled for, the correlation between coverage and fault detection drops to low or moderate, and measured against real post release defects it is statistically insignificant. Two well conducted studies find stronger links, and the disagreement is mostly about method. Coverage is a floor that flags an inadequate suite, not a target that proves quality.

Coverage is the number every organisation reports and the number a diligence engineer discounts. This post is the evidence for why, including the studies that cut the other way, because an article that showed only the convenient half would be committing the error it describes.

The anchor study, and what it actually measured

Laura Inozemtseva and Reid Holmes, at ICSE 2014, in a paper that won a Distinguished Paper award.

They generated 31,000 test suites of controlled, fixed sizes by randomly sampling test methods from the master suites of five large real Java systems, including Closure Compiler at up to 724,089 source lines. For each generated suite they measured statement, decision and modified condition coverage using CodeCover, and measured effectiveness as mutation score using PIT.

The central finding has to be stated two ways or it is misleading.

Ignore suite size, and coverage and effectiveness show a moderate to high correlation. This is the version everybody's intuition already holds, and it is not wrong as far as it goes.

Control for suite size, comparing only suites with the same number of tests, and the correlation drops to low or moderate. For Apache POI it fell from about 0.94 to somewhere in the range 0.46 to 0.85. For Joda-Time it fell essentially to zero.

The apparent relationship was largely an artifact. Bigger suites have more tests. More tests raise coverage and also catch more faults. Size was driving both, and once you hold it still, most of the relationship goes with it.

A second result from the same paper is underquoted and worth having. Statement, decision and modified condition coverage correlate 0.91 to 0.92 (Kendall's tau) with each other. They are largely measuring the same thing, so buying a stronger coverage criterion does not buy proportionally more insight into your suite.

Their own conclusion, close to the original wording: coverage, while useful for identifying under tested parts of a program, should not be used as a quality target, because it is not a good indicator of test suite effectiveness.

The study that used real bugs

The obvious objection to the above is that effectiveness was measured with mutants, which are injected faults rather than real ones.

Kochhar, Lo, Lawall and Nagappan did the harder version, published in IEEE Transactions on Reliability in 2017. 100 large Java projects, and the dependent variable is real post release defects: the bugs that actually reached users.

Coverage's correlation with those defects was statistically insignificant at the project level. Spearman rho of about -0.059, p = 0.559. There was no significant correlation at the file level either.

That is the single strongest number available for the claim that coverage does not predict the bugs that reach production, and it is strongest precisely because it does not depend on believing that mutants stand in for real faults.

The studies that push back

Both of these are good work and both cut against the above. Reporting them is not a hedge; it is the difference between a defensible position and a slogan.

Gligoric and colleagues, ISSTA 2013. Comparing non-adequate test suites using coverage criteria, they found Kendall tau between 0.452 and 0.757 between coverage and mutation score across coverage types. That is a real relationship. Coverage carries signal, particularly when what you are doing is comparing two inadequate suites against each other.

Gopinath, Jensen and Groce, ICSE 2014. Same conference as the anchor study, opposite lean. They measured around 200 developer written suites rather than randomly sampled ones, and found coverage is correlated with effectiveness for all coverage types. More pointedly, they found that adding suite size to the statistical model did not improve its fit, which attacks the size confound argument directly.

Inozemtseva and Holmes note this may reflect multicollinearity: size and coverage are entangled tightly enough in real suites that a model may be unable to separate their contributions. That is a reasonable response and it is not a refutation.

What the disagreement is actually about

Three methodological differences explain most of it.

Randomly sampled suites against developer written ones. The anchor study built suites by sampling test methods at random. Real suites are not random; developers write tests aimed at things, so a developer written suite of a given coverage may be systematically better than a random one of the same coverage.

Mutants against real bugs. Kochhar's real-defect result is the strongest single number, and it is also the noisiest measurement, because post release defect counts depend on how a project reports bugs, how much it is used, and by whom.

Whether size is controlled. This is the crux, and Gopinath's finding that size adds nothing to the model is the sharpest challenge to the whole framing.

The position that survives all three: low coverage reliably tells you a suite is inadequate. High coverage tells you nothing on its own. Coverage is a floor, not a target.

Do not say coverage is worthless. The evidence does not support it, and the overclaim hands anyone who has read Gligoric or Gopinath a reason to dismiss the entire argument.

The study that points at the answer

There is a fourth result that is easy to miss because it qualifies rather than supports.

Chekam, Papadakis, Le Traon and Harman, at ICSE 2017, built a methodology that avoids the clean program assumption, a known bias in earlier coverage studies. They found that only high levels of strong mutation coverage have strong fault revealing power, while statement, branch and weak mutation coverage have little.

That is the bridge out of this whole argument. Coverage is weak, and there is a better signal available, and it is the one that asks whether your suite catches a fault you deliberately introduced.

How to use coverage correctly

The standard that governs software whose failure kills people already uses it the right way, and has for decades.

DO-178C requires requirements based testing first. Structural coverage is then measured to reveal code that no requirements based test exercised. Coverage is a gap finder pointed at the tests, not a quality score pointed at the code.

That is precisely what Inozemtseva and Holmes recommend, arrived at independently by a certification body. Read your coverage report for the parts that are near zero, because those tell you something real. Ignore the headline percentage, because it does not.

And then ask the next question, which is not how much of the code ran but what was checked about it. That is in coverage measures what ran, not what you checked, and the playable version, where two suites both report one hundred percent and catch wildly different numbers of injected faults, is the coverage against mutation tool.

FAQ

Does code coverage predict bugs? Weakly at best. Once suite size is controlled for, the correlation with fault detection drops to low or moderate, and measured against real post release defects across 100 Java projects it is statistically insignificant, at Spearman rho of about -0.059 with p of 0.559.

Is 100 percent code coverage worth chasing? Not as a target. Coverage is a floor: low coverage reliably flags an inadequate suite, while a high number guarantees nothing about whether the executed code was actually checked.

Do any studies disagree? Yes, two credible ones. Gligoric et al. found Kendall tau of 0.452 to 0.757 between coverage and mutation score, and Gopinath et al. found coverage correlated with effectiveness on around 200 developer written suites, with suite size adding nothing to the model. The disagreement is mostly about method.

Does the type of coverage matter? Barely. Statement, decision and modified condition coverage correlate 0.91 to 0.92 with each other, so a stronger criterion is largely measuring the same thing.

What should we measure instead? Whether the suite catches faults you inject deliberately, and how much it asserts. Assertion density predicts effectiveness more strongly than coverage, and mutation detection tracks real fault detection more strongly than statement coverage does.