A Test That Has Never Failed Is a Decoration

Mostafa Dhouib·18 min read

A Test That Has Never Failed Is a Decoration

The short answer. Passing is a property of the test, not of the code, until you have watched that test fail on a known defect. A clean demo and a green pipeline are the same. The only credible evidence any check works is a negative control: deliberately break the thing it guards and confirm the check fails.

The validator passed

On 19 July 2024, a content update crashed roughly 8.5 million Windows machines. The interesting part is not the crash. It is that the file had been through a validator, and the validator approved it.

CrowdStrike's own root cause analysis lays it out. The sensor's IPC Template Type defined twenty one input parameter fields. The Content Interpreter supplied twenty. That mismatch had been sitting there since February, and nothing had ever noticed, because every Template Instance shipped until then used a wildcard on the twenty first field. The wildcard meant the twenty first input was never read against the array, so the mismatch could not manifest. Every deployment succeeded.

Then an instance shipped with a non wildcard match on that field, the interpreter reached past the end of a twenty element array, and the exception took the kernel with it.

The Content Validator contained a logic error: it passed the instance on the assumption that twenty one inputs would be available at runtime, when only twenty were. Read that again slowly, because the shape of it is the whole subject. The check was not missing. It was not skipped. It ran, it returned green, and its green said nothing whatsoever about the property that failed.

The RCA is blunt about the record of success too. The mismatch was not discovered during the sensor release testing process, the template type stress testing, or the first several successful deployments. A history of green was accumulating the entire time, and it was worth nothing, because the failure path had never been driven.

Passing is a property of the test

Here is the claim this whole piece rests on.

A test that has never failed has never been shown to test anything. Its green tells you the test passed. Whether it would have gone red on a broken system is a separate question that you have not asked, and until you ask it, you have a decoration: something that looks like a control and functions as reassurance.

Heartbleed is the same shape in a different domain. OpenSSL's heartbeat handler read a client supplied length field and copied that many bytes back, without checking the declared length against what the client actually sent. A client could claim sixty four kilobytes while sending a few, and the server would hand back sixty four kilobytes of adjacent process memory: private keys, session data, credentials. RFC 6520 already required that such a message be discarded silently. The check was simply absent.

And the tests passed. They passed because they only ever exercised well formed heartbeats, where the declared length equalled the actual length. It was not a weak suite for the inputs it covered. It never supplied the one input class that mattered. The eventual fix, in the words of the people who shipped it, was as trivial as the bug: a value check.

Both of these are usually filed under "they should have tested more." That is the wrong lesson and it is an expensive one, because it sends teams to write more tests of the kind they already have. The right lesson is narrower: nobody had ever watched those checks fail.

Lie two: high coverage means the code is safe

This is where most engineering organisations think they have the problem solved, and where the evidence is least comfortable.

Inozemtseva and Holmes generated thirty one thousand test suites of controlled sizes across five large Java systems, including Closure Compiler at up to 724,089 source lines, and measured coverage three ways against a mutation based measure of effectiveness. Ignore suite size and coverage looks moderately to highly correlated with effectiveness. Control for suite size, comparing suites with the same number of tests, and the correlation drops to low or moderate. For Apache POI it fell from about 0.94 to somewhere in the range 0.46 to 0.85. For Joda-Time it fell to essentially zero.

The apparent relationship was largely size wearing a disguise. Bigger suites have more tests, more tests raise coverage, and more tests catch more faults.

Kochhar and colleagues then did the harder version of the study, using real post release defects across a hundred large Java projects instead of injected ones. Real bugs, the ones that reached users. The correlation with coverage was statistically insignificant: Spearman rho of about -0.059, p of 0.559, with nothing significant at the file level either.

Inozemtseva and Holmes also found that the type of coverage barely matters. Statement, decision and modified condition coverage correlate 0.91 to 0.92 with each other. They are largely measuring the same thing, so buying a stronger criterion does not buy proportionally more insight. Their own conclusion is worth quoting close to the original: coverage, while useful for identifying under tested parts of a program, should not be used as a quality target, because it is not a good indicator of test suite effectiveness.

What the disagreement actually is

An article about the danger of showing only the flattering evidence does not get to show only the flattering evidence. So here is the other side, in full.

Gligoric and colleagues found Kendall tau between 0.452 and 0.757 between coverage and mutation score across coverage types. That is a real relationship. Coverage is not nothing, and it carries genuine signal when comparing suites that are not adequate.

Gopinath, Jensen and Groce, at the same conference as the anchor study and leaning the opposite way, measured around two hundred developer written suites rather than randomly sampled ones and found coverage correlated with effectiveness for every coverage type. They also found that adding suite size to the model did not improve its fit, which is the direct challenge to the size confound argument. Inozemtseva and Holmes note this may reflect multicollinearity, since size and coverage are entangled tightly enough that a model may not be able to separate them.

The disagreement is real and it is mostly about method: randomly sampled suites against developer written ones, injected mutants against real bugs, size controlled or not.

So do not say coverage is worthless. Say the thing that survives all of it, which is that coverage is a floor, not a target. Low coverage reliably tells you a suite is inadequate. High coverage tells you nothing on its own. This is also, incidentally, exactly how DO-178C uses it: requirements based testing first, then structural coverage measured to find code that no requirements based test exercised. A gap finder against the tests, not a quality score.

Lie three: more tests mean more confidence

More tests help if they assert on behaviour. A test can execute a line and check nothing about it, and that line is now covered and completely unprotected. The coverage tool cannot tell the difference, and neither can the report it produces.

Zhang and Mesbah found that assertion count and assertion density predict fault detection effectiveness more strongly than coverage does. That is the mechanism underneath the whole coverage result. Coverage measures execution. Effectiveness comes from assertion.

If you want the direct measure rather than a proxy for it, the question is whether the suite catches a fault you deliberately put there. Just and colleagues examined 357 real faults across five applications totalling 321,000 lines of code, against 230,000 mutants, controlling for statement coverage. A coupling relationship held for 73 percent of the real faults: a test that caught the real bug also killed at least one mutant. Mutant detection correlated with real fault detection more strongly than statement coverage did.

One number from that study is worth carrying on its own. Of 480 triggering test pairs, adding the fault detecting test raised mutation score in 75 percent of cases and raised statement coverage in only 46 percent. The test that catches the bug frequently adds no coverage at all.

Mutation testing has a real ceiling and it should be stated. Twenty seven percent of those faults coupled to no generated mutant, and 17 percent couple to none even in principle under common operators, dominated by algorithm rewrites, code deletion and numerical handling. It is the best available proxy, not truth.

Lie four: it works in the demo

A demo is a happy path performance. As evidence it is close to worthless, and its persuasive power is roughly inverse to its information content.

Ariane 5, flight 501, 4 June 1996. The inertial reference software was reused unchanged from Ariane 4 and had a flawless record across every prior flight and demonstration. Inside it, an alignment function that served no purpose after liftoff on Ariane 5 converted a 64 bit floating point horizontal bias value to a 16 bit signed integer.

Ariane 5's trajectory produced horizontal velocity values roughly five times larger than Ariane 4's. The value exceeded 32,767, the conversion overflowed, and the resulting Operand Error was interpreted as a hardware failure. The active unit shut down. The backup had already failed microseconds earlier for precisely the same reason, being the same software running the same data. With no attitude data the launcher commanded full nozzle deflection, tore apart under aerodynamic load, and self destructed. Roughly 370 million dollars of payload.

Only four of seven at risk conversions had been protected against overflow. This one was left unprotected to hold the unit within a workload target of about 80 percent.

The Inquiry Board's finding on verification is the sentence to remember: the code was used with no specific verification for Ariane 5. One test driving the module with the actual Ariane 5 trajectory forces that overflow on the ground, at a bench, for the cost of an afternoon.

The demo record was not weak evidence about Ariane 5. It was evidence about Ariane 4. A module's test history is a statement about the environment it ran in, and reuse moves the module while leaving the evidence behind.

Lie five: green CI means the system is healthy

At scale, green and red stop meaning anything without flakiness control, and the numbers here are not marginal.

Google runs on the order of 4.2 million tests in continuous integration. Almost 16 percent of them show some level of flakiness. 84 percent of transitions from pass to fail are flaky rather than real regressions, so the large majority of red events are not telling you anything happened. And the one that should stop a room: only 1.23 percent of tests have ever found a breakage.

That is not a Google problem. Lam and colleagues measured about 4.6 percent of tests flaky across five Microsoft projects, and found that of 311 build failures, 97 could not be reproduced locally even with a hundred reruns. Facebook's stated position is to assume all tests are flaky and engineer a statistical signal rather than trust a single run.

The compute cost of re-running enormous suites is the obvious cost. The dangerous one is trust. When red is usually noise, engineers do the rational thing and start ignoring red, quarantining tests, re-running until green. A quarantined test is a disabled safety control. A real regression hiding behind a flaky one reaches production wearing a green badge.

Knight Capital is the organisational version of this, at a different layer. On 1 August 2012 a deployment reached seven of eight servers, and a repurposed feature flag reactivated dormant routing code on the eighth. Roughly four million executions across 154 stocks in about 45 minutes, roughly 460 million dollars. Two things are worth separating. There was no version consistency check that would have gone red when the servers diverged, so one control was simply absent. But 97 automated pre open warning emails did fire, referencing the dormant code by name, and were not escalated. An alarm that nobody acts on and a test that nobody believes are the same failure.

The reframe: a negative control

Everything above collapses into one move.

Before you trust a green check, make it go red on a known broken input. Break the thing it guards, confirm the check fails, restore.

The term is borrowed from the lab, where a negative control is the sample that must come back negative. If it does not, the finding is that your assay is wrong, not that your sample is unusual. Software has the same need and mostly lacks the habit.

In software the systematic form is mutation testing: inject a small fault, confirm the suite kills it. A surviving mutant is a located test gap, which is a far more useful object than a percentage, because it names the assertion you do not have. In firmware the form is fault injection: corrupt a sensor value, force an error return, flip a bit, drop a packet, delay an interrupt, and confirm the system notices.

None of this is exotic. ISO 26262 makes fault injection highly recommended at ASIL D, and requires the single point fault metric to reach 99 percent, assessed through fault injection campaigns. DO-178C requires MC/DC at Level A, which is the formal version of exactly this argument: exercising a condition is not the same as proving it does anything. Google runs mutation testing across a roughly two billion line monorepo, diff based on changed covered lines, one mutant per line, surfacing survivors at code review.

Google's stance on the number is worth borrowing too. They do not compute a classic mutation score at all, and they optimise the usefulness of surfaced mutants rather than a kill percentage, having driven the unproductive rate from about 80 percent down to about 15. Chasing 100 percent is not available anyway: equivalent mutants produce behaviourally identical programs that nothing can kill, and their prevalence across the literature runs 4 to 39 percent.

The negative control has to live where the assumption lives

This is where the discipline gets misapplied, so it is worth being precise.

Mars Climate Orbiter was lost because a ground software module produced thruster impulse in pound force seconds while the interface specification required newton seconds. Navigation consumed the value as newton seconds, underestimating the effect by a factor of 4.45, and the spacecraft entered the atmosphere far below its survivable altitude. Both sides were internally correct. No unit test inside either module would have found it, because the defect lived in the agreement between them. The negative control belonged at the boundary: assert the units at the handoff, or run a physical plausibility check on the result.

Two further cases need more care than they usually get.

MCAS depended on a single angle of attack sensor for a function Boeing's own safety assessment classified as hazardous, a classification that normally precludes single sensor dependence. The certification analysis also understated the system's authority by more than four times and did not account for repeated re triggering. The reports locate the cause in system safety assessment and requirements, not in a missing unit test, and it would misrepresent them to say otherwise. What is true is narrower and still useful: a fault injection test feeding a single erroneous sensor value exposes both the dependence and the runaway behaviour. The negative control had to exist at the system level.

The Therac-25 is the same caution. It removed the hardware interlocks that its reused code assumed were present, and a race condition reachable by fast operator edits, combined with a one byte counter overflowing to zero, bypassed a safety check. Six overdoses, three deaths. Leveson and Turner conclude that the accidents came from systemic engineering and institutional failure rather than a single line of code, and that early incident reports were dismissed. A stress test scripting rapid edits surfaces the race. It does not fix an organisation that does not believe its own incident reports.

So: the discipline scales, from a unit assertion to an interface assertion to a hazard level fault injection campaign. And on its own it is necessary rather than sufficient. Heeding the red is the other half, and Knight's 97 emails are what its absence looks like.

The one line you can run this week

Take the module you would least like to be silently wrong about. Money, safety, access control, or the boundary where you hand a value to somebody else.

Delete one line of production code, or flip one condition from == to !=, or < to <=, or && to ||. Run your suite.

If something goes red, that check works and you now have evidence instead of a feeling. If nothing goes red, that part of your suite is a decoration, and you have learned exactly which assertion is missing. Revert either way. The whole exercise takes minutes and needs no tooling.

It is a manual one mutant mutation test, which is to say it is the smallest possible instance of the only discipline in this article. It is also the version that would have caught CrowdStrike's field count, Heartbleed's missing bounds check, and Ariane's unprotected conversion.

What good looks like

If you want the standard rather than the argument, it is nine lines. Tick one only where somebody has actually watched the check go red.

Every guard, alarm and limit has a test that drives the bad input and asserts the guard fires, not merely a test confirming good input passes. CI contains at least one deliberately broken input or build that must fail, as a meta test on the harness itself. The surviving mutant list is reviewed on critical paths rather than a coverage percentage. Flaky tests are fixed, or quarantined with a ticket and a deadline, never silently re-run to green. Reused code is re-verified against the new operating envelope rather than its old one. Units and interfaces are asserted at every team and contractor boundary. Every parser, protocol and sensor input path has a fuzz test and a negative test. Deployments have a version consistency check that goes red on divergence. Safety relevant single points of failure are identified and fault injected.

Each of those lines has a public failure behind it, which is what makes them arguable in a room rather than a matter of taste.

And the honest summary of the evidence, since this piece has leaned on it heavily: rigor pays and rigor is not free. Four industrial teams saw pre release defect density fall between 40 and 90 percent under test driven development, at a subjective cost of 15 to 35 percent more initial development time. Those are case studies with self selected teams and no randomisation. Directionally strong, causally soft, and worth quoting with both halves attached.

The cheapest item on the list has no measured cost at all. Delete a line, and see whether anything notices.

FAQ

Does passing tests mean my code works? No. Passing is a property of the test until somebody has watched that test fail on a known defect. The CrowdStrike Content Validator passed the file that crashed roughly 8.5 million machines, and Heartbleed's suite passed because it only ever exercised well formed heartbeats. In both cases the check ran, returned green, and said nothing about the property that failed.

How do I know if my tests actually work? Delete one line of production code, or flip one condition, and run the suite. If nothing goes red, that part of the suite is a decoration and you have found the missing assertion. It is a manual one mutant mutation test, it needs no new tooling, and it runs in minutes.

Is code coverage a good quality target? It is a floor rather than a target. Once suite size is controlled for, coverage correlates only weakly with fault detection, and against real post release defects the correlation is statistically insignificant at Spearman rho of about -0.059. Low coverage reliably flags an inadequate suite; high coverage guarantees nothing. Note that Gligoric et al. and Gopinath et al. both found stronger links than the anchor study, so the honest position is the floor framing rather than the claim that coverage is worthless.

What is a negative control in software testing? Deliberately breaking the thing a check guards and confirming the check fails, then restoring. The systematic form in software is mutation testing; in firmware it is fault injection. It is what the highest assurance standards already mandate, with MC/DC required at DO-178C Level A and fault injection highly recommended at ISO 26262 ASIL D.

Where should the negative control live? At the level the assumption lives at. A units mismatch between two teams is invisible to any unit test inside either module and needs an assertion at the boundary, which is the Mars Climate Orbiter lesson. A single sensor dependence needs fault injection at the system level, which is the MCAS lesson. A unit level habit alone will not reach either.