Green CI Is Lying to You: The Flaky Test Problem
Mostafa Dhouib··6 min readAbout 16 percent of Google's 4.2 million tests are flaky and 84 percent of pass to fail transitions are not real regressions. Only 1.23 percent of tests have ever found a breakage. The compute cost is the obvious one; the trust cost is the dangerous one, because a quarantined test is a disabled safety control.
Green CI Is Lying to You: The Flaky Test Problem
The short answer. At scale, green and red stop carrying information. About 16 percent of Google's 4.2 million tests are flaky, 84 percent of pass to fail transitions are flaky rather than real, and only 1.23 percent of tests have ever found a breakage. Teams rationally learn to treat red as noise, and a real regression then ships green.
Most arguments about flaky tests are about developer annoyance and CI compute bills. Those are real and they are not the problem. The problem is that flakiness destroys the information content of your pipeline, and it does it quietly.
The numbers
From John Micco's ICST 2017 keynote on the state of continuous integration testing at Google, and the Google Testing Blog. The primary is a talk rather than an archival paper, which is worth stating, though the figures are corroborated widely enough in peer reviewed work to rely on.
Google runs on the order of 4.2 million tests in continuous integration.
Almost 16 percent of them show some level of flakiness.
84 percent of transitions from pass to fail are from flaky tests. So when a build goes red, the overwhelmingly likely explanation is that nothing happened.
Roughly 1.5 percent of all individual test runs report a flaky result.
And the one that should stop a room: only 1.23 percent of tests have ever found a breakage. Nearly ninety nine percent of that suite has never once done the thing a test exists to do.
Of 115,160 test targets that had both passed and failed over a window, 41 percent were flaky.
It is not a Google problem
Two independent measurements at two other organisations of comparable scale.
Microsoft. Lam, Muslu, Sajnani and Thummalapenta, at ICSE 2020, measured about 4.6 percent of tests flaky across five Microsoft projects using the CloudBuild system. The detail that matters most in that paper: of 311 cloud build failures examined, 97 could not be reproduced locally even with 100 reruns. Flakiness is not simply "run it again."
Facebook. Harman and O'Hearn's stated position is Assume All Tests Are Flaky: design CI as though any given red might be noise and engineer a statistical signal out of many runs, rather than trusting a single one.
Academic baselines put ordinary projects lower, around 0.5 to 1 percent, which makes sense. Scale amplifies rare nondeterminism, so the hyperscaler numbers are an early view of where a growing system is heading rather than a different phenomenon.
Why tests go flaky
Luo, Hariri, Eloussi and Marinov, at FSE 2014, analysed 201 commits that fixed flaky tests across 51 open source projects and categorised the root causes.
Async Wait leads at roughly 45 percent of flaky test fixes: a test that waits a fixed duration for something asynchronous, and the duration is usually right. Then concurrency, then test order dependency, where a test passes alone and fails after another test has run.
None of these is exotic and all of them are the ordinary consequence of testing systems that have real timing in them.
The cost that actually hurts
The compute cost of re-running enormous suites is the visible one, and it is the one that gets budgeted.
The trust cost is the one that kills.
When red is frequently noise, engineers start ignoring red. They re-run until green, they quarantine the offenders, they stop reading the notification channel. Every one of those responses is rational, given the base rate. If 84 percent of red is meaningless, treating red as meaningless is a correct inference from the evidence available.
And then a real regression arrives, hides behind a flaky test, and reaches production wearing a green badge.
A quarantined test is a disabled safety control. That is the sentence worth carrying out of this post. Quarantine is not a neutral holding state; it is switching off a check, and it is usually done without an owner, a deadline, or a record.
The same failure, in a different channel
This exact pattern destroyed a firm.
At Knight Capital on 1 August 2012, 97 automated pre-open warning emails fired, naming the dormant code that was about to cost roughly 460 million dollars. They were not escalated and not acted upon.
That is not a technology failure and it is not carelessness. It is the same inference: a channel that habitually produces output requiring no action trains its readers to skip it. The 97 emails and a 16 percent flaky rate are the same artifact at different layers of the stack.
Which means the fix is the same too, and it is unglamorous. Reduce the volume until the signal is credible. Anything you cannot fix gets an owner and a deadline rather than a folder.
What to do
Measure your flaky rate. Tests with inconsistent results over 30 days divided by total. Most teams have never computed this and are surprised by it. If it exceeds a few percent, treat it as a strategic risk rather than an annoyance, because at that level your pipeline has stopped being evidence.
Give every quarantine a ticket and a deadline. A quarantine without an owner is a permanently disabled control, and it will still be disabled in two years.
Keep a flaky test ledger. This is also one of the four artifacts a diligence engineer asks for, and having one signals that red still carries information in your organisation. Founders often assume admitting to flaky tests looks weak. It is the opposite: everyone at scale has them, and only some people can tell you which.
Fix Async Wait first. It is 45 percent of the problem and it is usually a fixed sleep that should be a condition.
The general argument is in a test that has never failed is a decoration, and the organisational version of the same failure is in Knight Capital. To score your own controls, including the flaky test ledger, use the verification diligence scorecard.
FAQ
How common are flaky tests? About 16 percent of Google's 4.2 million CI tests show some flakiness, and about 4.6 percent across five Microsoft projects. Ordinary projects sit closer to 0.5 to 1 percent, because scale amplifies rare nondeterminism.
What percentage of CI failures are real? At Google, 84 percent of transitions from pass to fail come from flaky tests, so the large majority of red events are not regressions.
How many tests ever catch anything? At Google, 1.23 percent of tests have ever found a breakage. Nearly the entire suite has never once done the thing a test exists to do.
Why is quarantining a flaky test dangerous? Because a quarantined test is a disabled safety control. Without an owner and a deadline it stays disabled indefinitely, and a real regression can hide behind it and reach production.
What causes flaky tests? Async Wait leads at roughly 45 percent of fixes, followed by concurrency and test order dependency, across 201 flaky test fix commits in 51 open source projects.
Nine controls, each tied to a public failure that cost somebody nine figures or a life, with the four artifacts a verification data room holds and the one question an investor should ask in the room. Built to be printed and ticked only where somebody has actually watched the check go red.
Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.
Bring us the program