Coverage Measures What Ran, Not What You Checked

Mostafa Dhouib·4 min read

Coverage Measures What Ran, Not What You Checked

The short answer. Coverage records execution. Effectiveness comes from assertion. A test can run a line and check nothing about its behaviour, which counts as covered and is entirely unprotected, and the report cannot tell the two apart. Assertion count and density predict fault detection more strongly than coverage does.

The research says coverage predicts fault detection weakly. This post is the reason why, and it is a one sentence mechanism that most teams can act on the same afternoon.

The finding

Zhang and Mesbah, at FSE 2015, in a paper whose title is the result: assertions are strongly correlated with test suite effectiveness.

Assertion count and assertion density are strong predictors of a suite's fault detection effectiveness, and stronger predictors than coverage.

That is the whole study for our purposes, and it explains rather than merely reports. It is the mechanism the coverage results have been circling.

Why this resolves the confusion

Coverage instruments execution. When a line runs, the counter increments. That is the entire measurement.

Effectiveness comes from assertion: from a test having stated what the behaviour should be, so that a change in behaviour causes a failure.

These are different properties and nothing forces them to coincide. A test can execute a line and assert nothing about it. The line is now covered and completely unprotected. The coverage tool records it identically to a line guarded by three careful assertions, because from the tool's position they are identical: both ran.

This is why two suites with the same coverage number can differ enormously in what they would catch, and why the coverage percentage carries so little information on its own.

The shapes this actually takes

It is worth naming them, because they are common and none of them looks negligent while you are writing it.

A test that calls a function and asserts only that it did not throw. Extremely common in integration tests. It covers everything the call touches and protects none of it.

A snapshot or golden file test that was regenerated after the behaviour changed. This is the subtle one. The test now asserts that the new behaviour is the new behaviour. It will catch a future regression and it silently ratified the change that prompted the regeneration, which may have been the bug.

An assertion on shape rather than on value. The response is an object with three keys. The list has five elements. The type is a float. All true, all covered, and none of them would notice a units error, a sign flip, or a factor of 4.45.

Setup and teardown counted as covered. It ran, because it always runs. It is covered in every report and asserted by nothing.

Logging and trivial getters. Which is exactly why Google suppresses these as arid nodes when generating mutants: mutating them produces findings nobody can act on. That suppression is what took their unproductive mutant rate from roughly 80 percent down to roughly 15.

What to do about it

Read assertion density alongside coverage. A file with high coverage and near zero assertions is the specific failure this post names, and it is visible with a rough count.

Better, use the direct measure. Inject a fault into a covered line and see whether anything goes red. That answers the question coverage structurally cannot, because it changes the behaviour rather than the execution. The manual, no-tooling version is to delete a line or flip one condition and run the suite.

On critical paths, review the surviving mutant list rather than the percentage. Each survivor names an assertion you do not have, in a specific place, which is a far more useful object than a number that went from 84 to 86.

The general argument is in a test that has never failed is a decoration, and the evidence about coverage itself is in what the research actually says. If you want to watch this happen on a decision from your own code, with both suites reporting one hundred percent and catching different numbers of faults, that is the coverage against mutation tool.

FAQ

What is the difference between code coverage and assertions? Coverage records which lines executed. Assertions record which behaviour was checked. A line can execute with nothing asserted about it, and the coverage report cannot distinguish that from a line guarded by careful assertions.

Why does high coverage still miss bugs? Because covered means executed, not checked. Effectiveness comes from assertion, and assertion count and density predict fault detection more strongly than coverage does.

Is assertion density a better metric than coverage? It is a stronger predictor, so it is a better proxy. The direct measure is better still: inject a fault and see whether the suite kills it.

What kinds of test look covered but protect nothing? Tests that assert only that no exception was thrown, snapshot tests regenerated after a behaviour change, assertions on shape rather than value, and setup code that is covered because it always runs.

How do I find covered but unasserted code? Flip a condition or delete a line in the file and run the suite. If nothing goes red, that region is covered and unprotected, and you have just located the missing assertion.