Tagged verification
30 pieces
- · 10 min read
Accept The Work, Not The Claim: An Agent's Own Status Is Not Evidence
A headless agent run that hits a rate limit can exit with a success status and a zero exit code, having written nothing at all. Every reliability feature that reads the agent's own account of what happened, retry, recovery, the status pane, then repeats that false claim instead of catching it. The only fix is an independent check between running and accepting.
- · 10 min read
Plant the Faults So the Harness Proves It Can See Them
A harness that has never detected a known fault has not been shown to detect anything. Injecting deliberate, known faults is the step that converts a green suite into a suite that can fail for the right reasons. This covers the fault classes worth planting in an extraction or retrieval pipeline, how each one should present, and why a planted fault is the positive half of a pair whose other half is the negative control.
- · 10 min read
What Should Never Be An Agent: Use The Benchmark Gap As Your Decision Criterion
Published benchmarks of agents on realistic computer-use tasks report completion rates in the low tens of per cent, while humans on the same tasks sit near ceiling. That gap is not a reason to avoid agents. It is the input to a decision procedure: a low success rate is fine when checking the answer is cheaper than producing it, and unacceptable when the action is irreversible or verification costs more than doing the work yourself.
- · 10 min read
Every Risk Control Is Verified Twice
The governing international standard for medical device risk management requires each risk control to be verified twice: once that it was implemented, and once that it is effective. Those are two different tests. A plan that writes only the first has done half the work, and effectiveness is the harder half.
- · 8 min read
A Rule With No Instance Is Not Found by Inspection
A business rule that only fires on a condition absent from your copied data will not be discovered by reading the schema, however carefully you read it. That is the structural limit of static discovery, and it is why keeping the old system running during replacement matters more than the completeness of the catalogue. Shadow running and output comparison on live traffic are the only mechanisms that close the gap.
- · 5 min read
IEC 62304: Verification Scales With Risk Class
Class A is no possible injury, Class B non-serious injury, Class C death or serious injury. Verification obligations rise with the class: more rigorous unit, integration and system testing, documented procedures, and traceability from software requirements through tests to risk controls under ISO 14971.
- · 5 min read
ISO 26262 Mandates Fault Injection at ASIL D
Branch coverage is highly recommended at ASIL B, C and D; MC/DC and fault injection become highly recommended at ASIL D. The hardware side permits only about 1 percent residual single point faults at ASIL D, assessed through fault injection campaigns. Negative control discipline, written as a requirement.
- · 5 min read
What DO-178C and MC/DC Actually Require, With a Worked Example
Branch coverage on D = A and B is satisfied by two tests, and a bug that ignores B passes both. MC/DC requires each condition to independently flip the outcome, needing three tests, and it catches that bug. It is the cleanest illustration of covered against proven.
- · 6 min read
Every Reused Module Is Untested Until You Run It Against the New Envelope
A reused module carries a correctness argument scoped to its old environment. Ariane 5 lost a launcher to a conversion that was safe within Ariane 4's velocity range, and the Therac-25 killed three people reusing code whose assumption about hardware interlocks was no longer true.
- · 5 min read
Fuzzing and Sanitizers for Every Input Path
A coverage guided fuzzer mutates inputs to drive new paths, and a sanitizer catches the out of bounds read that a functional test never notices. Neither half is sufficient alone, and together they cover the adversarial input class where Heartbleed lived.
- · 5 min read
Property Based Testing: Stop Hand Picking the Inputs That Pass
Instead of hand picking examples, state a property such as declared length never exceeds actual length, and let the framework generate hundreds of inputs and shrink any failure to a minimal case. Heartbleed is exactly the class of bug an implementer's own examples systematically miss.
- · 5 min read
Fault Injection for Firmware: Corrupt the Sensor and Confirm It Notices
Corrupt a sensor value, force an error return, flip a bit, drop a packet, delay an interrupt, and confirm the system detects it. This is the MCAS lesson, ISO 26262 makes it highly recommended at ASIL D, and Renode and QEMU let you run the campaign without hardware.
- · 6 min read
Mutation Testing in Practice, Without It Taking All Night
Naive full repo mutation runs the suite once per mutant and takes hours. Google's method makes it tractable at roughly two billion lines: mutate only changed covered lines, one mutant per line, suppress arid nodes, and surface survivors at code review rather than computing a score.
- · 5 min read
The One Line Test That Proves Your Suite Works
Delete one line of production code, or flip one condition, and run your suite. If nothing goes red, that part of your suite is a decoration. It is a manual one mutant mutation test, it needs no tooling, it runs in minutes, and it is the discipline that would have caught CrowdStrike, Heartbleed and Ariane.
- · 6 min read
Negative Controls: Prove Your Check Can Fail Before You Trust Its Green
Before trusting any green check, make it go red on a known broken input. Three concrete forms: inject a known defect and confirm the suite fails, keep a must-fail case in CI as a meta test on the harness, and physically drive an out of range condition to confirm the alarm fires.
- · 5 min read
Does Test Discipline Actually Reduce Bugs? The Honest Numbers
Four industrial teams saw pre release defect density fall between 40 and 90 percent, at a subjective cost of 15 to 35 percent more initial development time. A controlled study of 24 professionals found an 18 percent higher pass rate at 16 percent more time. Directionally strong, causally soft, and not free.
- · 5 min read
Stop Chasing 100 Percent Mutation Score
A mutation that produces behaviourally identical code can never be killed, so it inflates the surviving count and makes 100 percent unreachable. Prevalence runs 4 to 39 percent of all mutants. Google does not compute a classic mutation score at all, and optimises the usefulness of surfaced mutants instead.
- · 6 min read
Do Injected Faults Track Real Bugs? The Case for Mutation Testing
Across 357 real faults and 230,000 mutants, a coupling relationship held for 73 percent of real faults, and mutant detection tracked real fault detection more strongly than statement coverage did. The honest limit is that 17 percent of real faults couple to no mutant even in principle.
- · 4 min read
Coverage Measures What Ran, Not What You Checked
A test can execute a line and assert nothing about it. That line counts as covered and is completely unprotected, and no coverage tool can tell the difference. Assertion density predicts fault detection more strongly than coverage does, which is the mechanism underneath the whole coverage debate.
- · 6 min read
Green CI Is Lying to You: The Flaky Test Problem
About 16 percent of Google's 4.2 million tests are flaky and 84 percent of pass to fail transitions are not real regressions. Only 1.23 percent of tests have ever found a breakage. The compute cost is the obvious one; the trust cost is the dangerous one, because a quarantined test is a disabled safety control.
- · 7 min read
High Coverage Does Not Mean Safe Code: What the Research Actually Says
Control for suite size and the correlation between coverage and fault detection drops to low or moderate. Against real post release defects it is statistically insignificant. Two credible studies push back, and the disagreement is mostly about method. The claim that survives all of it is that coverage is a floor, not a target.
- · 8 min read
Your Test Suite Is a Diligence Asset, or a Liability an Acquirer Will Find
To a diligence engineer, a suite nobody has watched fail is unproven, and unproven is indistinguishable from absent. Seven red flags reprice deals and each maps to a public failure. The founder move is a verification data room; the investor move is asking the target to make a check fail on camera.
- · 7 min read
Therac-25: When the Untested Assumption Is a Safety Architecture
The Therac-25 removed the hardware interlocks its reused code assumed and gave the job to software. A race condition reachable by fast operator edits, plus a one byte counter overflowing to zero, bypassed a safety check. Six overdoses, three deaths, and a cause the investigators located in the system rather than in a line.
- · 6 min read
Boeing 737 MAX and MCAS: The Single Sensor Nobody Fault Injected
MCAS depended on one angle of attack sensor for a function Boeing's own assessment classified as hazardous, a classification that normally precludes single sensor dependence. The safety analysis also understated the system's authority by more than four times. This is a system safety failure, not a missing unit test.
- · 6 min read
Heartbleed Was One Missing Negative Test
OpenSSL's heartbeat handler copied as many bytes as the client declared without checking that against what the client actually sent, returning up to 64 KB of adjacent memory. The tests only ever sent well formed heartbeats, so the failure path was never driven once, and the fix was as trivial as the bug.
- · 6 min read
Mars Climate Orbiter: Assert Your Units at Every Boundary
A ground software module computed thruster impulse in pound force seconds while the interface specification required newton seconds. Navigation consumed it as newton seconds, underestimating the effect by a factor of 4.45, and no test or interface check at the contractor boundary ever reconciled the two.
- · 6 min read
Knight Capital: A 460 Million Dollar Deploy With No Consistency Check
A technician copied new code to eight servers by hand and one did not receive it. A repurposed feature flag then reactivated dormant routing logic on that one server. Roughly 4 million executions in 45 minutes, and no control existed that would have gone red on the divergence.
- · 7 min read
Ariane 5: The Code Worked in Every Demo and Destroyed the Rocket
The inertial reference module was reused unchanged from Ariane 4 and had a flawless demonstration record. Ariane 5 produced horizontal velocity values roughly five times larger, a conversion to a 16 bit signed integer exceeded 32,767, and the launcher was destroyed 37 seconds after liftoff.
- · 7 min read
The CrowdStrike Outage Was a Validator That Did Not Test the Thing
The template defined twenty one input fields and the interpreter supplied twenty. Earlier instances wildcarded the twenty first, so the mismatch could not manifest and four months of deployments succeeded. The validator passed the file that broke it because it checked an assumption rather than the field count.
- · 18 min read
A Test That Has Never Failed Is a Decoration
A passing suite, a clean demo and green CI are evidence of nothing until somebody has watched the check go red on a known broken input. Passing is a property of the test, not of the code. The only credible proof any check works is a negative control: break the thing it guards and confirm the check fails.