Ground Truth

The One Line Test That Proves Your Suite Works

Mostafa DhouibMostafa Dhouib··5 min read
The short answer

Delete one line of production code, or flip one condition, and run your suite. If nothing goes red, that part of your suite is a decoration. It is a manual one mutant mutation test, it needs no tooling, it runs in minutes, and it is the discipline that would have caught CrowdStrike, Heartbleed and Ariane.

The One Line Test That Proves Your Suite Works

The short answer. Delete one line of production code, or flip one condition from == to !=, < to <=, or && to ||, and run your test suite. If nothing goes red, that part of your suite is a decoration. It needs no new tooling, runs in minutes, and is the smallest possible proof that a check can fail.

Everything else in this pillar is an argument. This is the thing to do this afternoon.

The check

Take one line of production code. Delete it, or flip one condition in it. Run the suite. Revert.

The gate
Delete one line of production code, or flip one condition, and run your suite.
Something goes red
That check works. You have evidence rather than a feeling. Revert and move on.
Nothing goes red
That part of the suite is a decoration, and you now know exactly which assertion is missing.
This is the discipline that would have caught CrowdStrike, Heartbleed and Ariane 5.
FigureThe smallest possible proof that a check can fail. It is a manual one mutant mutation test, it needs no tooling, and it runs in minutes on the module you would least like to be wrong about.

That is all of it.

How to choose the line

This part decides whether the exercise is useful or theatre.

Not the easiest line to reach. The instinct is to pick something you know is well tested, which produces a green result that tells you what you already believed.

Pick the line you would least like to be silently wrong. A useful prompt: if this line were subtly incorrect, and nothing crashed, how long before anybody noticed and what would it cost by then? The line with the worst answer is the line to break.

In practice that is a pricing calculation, a permission check, a limit comparison, a unit conversion at a boundary, or the guard clause on a path that does something irreversible.

Change behaviour, not structure. Deleting a log statement or a comment proves nothing. You want the program to compute a different answer, so that a test asserting on the answer has something to object to.

Do it on a branch, and revert. The output of this exercise is the observation, not the change.

How to read the result

Something goes red. That check works, for that line. You now have evidence rather than a feeling, and you know which test is carrying the weight. Note which one it was, because that is the test to be careful with in future refactors.

Nothing goes red. That part of your suite is a decoration. This is not a failure of the exercise, it is the exercise succeeding: you have located a covered, unasserted line and you know exactly which assertion is missing. Write it, then repeat the break to confirm the new test catches it.

Something goes red for the wrong reason. Worth noticing separately. If a snapshot test fails because the output text changed, but no behavioural assertion fired, you have learned that the line is protected by coincidence rather than by intent. That protection will evaporate the next time somebody regenerates the snapshot.

Why this specific check

Because it is a manual, one-mutant mutation test, and mutation detection has been measured against real faults rather than assumed.

Just and colleagues, at FSE 2014, found a coupling relationship for 73 percent of 357 real faults: a test detecting the real bug also killed at least one mutant. And mutant detection tracked real fault detection more strongly than statement coverage did, controlling for coverage.

The specific operators that do most of that coupling work are conditional operator replacement, relational operator replacement and statement deletion. Which is exactly what this exercise does: flip a condition, change a comparison, delete a statement. The one-line self-check is not a folk practice that happens to resemble mutation testing. It is the highest yielding subset of it, done by hand.

It is also, concretely, the discipline that would have caught the three cases in this pillar with the clearest counterfactuals. CrowdStrike's field count. Heartbleed's missing bounds check. Ariane's unprotected conversion.

What it costs and what it does not

It costs a few minutes and it needs nothing you do not already have. No new tool, no CI change, no budget line, no approval.

That matters, because the honest evidence on the cost of test discipline is that it is not free. Four industrial teams saw pre-release defect density fall 40 to 90 percent, at a subjective cost of 15 to 35 percent more initial development time. Those are real numbers with real caveats, and they belong in a budget conversation.

This exercise sits outside that trade entirely. There is no version of the cost argument that reaches it.

What to do next, in order

This week. Run it on your most safety-relevant module. Then add one deliberately broken input to CI that must fail, as a meta test on the harness. Two afternoons, total.

The stop rule worth adopting. If more than zero mutations survive on a safety path, close them before shipping. Not a percentage target: a gate on a small set everybody can see.

This quarter. Stand up diff-based mutation testing on changed, covered lines for critical modules, using whatever your ecosystem has. Track the surviving mutant list, not a score, for reasons in stop chasing 100 percent mutation score.

The general argument is in a test that has never failed is a decoration, and the mechanism this exercise exploits is in coverage measures what ran, not what you checked.

FAQ

How do I check if my tests actually test anything? Delete one line of production code, or flip one condition, and run the suite. If nothing goes red, that part of the suite is not checking that behaviour.

What does it mean if nothing fails? You have found a covered but unasserted region: the suite executes that code and checks nothing about it. The coverage report cannot show you this, which is why the exercise exists.

Which line should I delete? The one you would least like to be silently wrong. A pricing calculation, a permission check, a limit comparison, a unit conversion at a boundary, or the guard on an irreversible action.

Why flip a condition rather than delete a log line? Because you need the program to compute a different answer. Structural changes that do not alter behaviour prove nothing about behavioural assertions.

Is this just mutation testing? It is a manual one-mutant mutation test, using the specific operators that do most of the coupling work with real faults: conditional replacement, relational replacement and statement deletion.

Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program