Ground Truth

Therac-25: When the Untested Assumption Is a Safety Architecture

Mostafa DhouibMostafa Dhouib··7 min read
The short answer

The Therac-25 removed the hardware interlocks its reused code assumed and gave the job to software. A race condition reachable by fast operator edits, plus a one byte counter overflowing to zero, bypassed a safety check. Six overdoses, three deaths, and a cause the investigators located in the system rather than in a line.

Therac-25: When the Untested Assumption Is a Safety Architecture

The short answer. The machine removed the hardware interlocks that its reused code assumed were present. A race condition reachable by fast operator edits, combined with a one byte counter overflowing to zero, bypassed a safety check. Leveson and Turner locate the cause in systemic engineering and institutional failure, not in one line.

Read this one carefully, or do not use it

The Therac-25 is the most misused case in software engineering. It gets deployed as an argument for unit tests, for type systems, for code review, for whatever the speaker was already advocating, and almost every one of those framings misrepresents the investigation it comes from.

Nancy Leveson and Clark Turner's study is explicit that the accidents came from systemic software-engineering and concurrent-design errors together with institutional failures, rather than from a single line of code. Their conclusion is a sentence about method: we must approach the problems of accidents in complex systems from a systems-engineering point of view and consider all contributing factors.

So the honest claim here is narrower than the usual one, and it is still worth making. The negative control has to exist at the level the assumption lives at. In this machine the load-bearing assumption was architectural, and no unit test could have reached it.

What actually happened

From Leveson and Turner, IEEE Computer 26(7), July 1993.

The Therac-25 removed the independent hardware interlocks that were present on the earlier Therac-20 and Therac-6, and relied on software to prevent unsafe beam configurations. It also reused code from those earlier machines. That code had been written under a safety assumption, that hardware would prevent unsafe states, which was no longer true.

A race condition existed between the operator-interface task and the treatment-control task. It could be triggered when an experienced operator edited treatment parameters quickly, within roughly an 8 second window, leaving the machine's software state inconsistent with its physical setup.

Separately, a one byte counter used during setup could overflow to zero. When it did so at the wrong moment, a safety check was bypassed, allowing the electron beam to fire in high energy mode without the beam-spreading target in place.

  1. Hardware interlocks removed, software given the job
    reused code still assumes they are there
  2. An experienced operator edits parameters quickly
    within roughly an 8 second window
  3. The interface task and the control task race
    software state inconsistent with the physical setup
  4. A one byte counter overflows to zero
    at the wrong moment, a safety check is bypassed
  5. The beam fires in high energy mode with no spreading target
  6. Six massive overdoses between 1985 and 1987, three deaths
    early incident reports were dismissed
Leveson and Turner locate the cause in systemic engineering and institutional failure, not one line of code.
FigureA system safety failure rather than a missing unit test. The reused software assumed hardware interlocks that the new machine no longer had, and the timing path that exposed it was never exercised.

Between 1985 and 1987 there were six massive radiation overdoses and three deaths.

On verification, the manufacturer relied almost exclusively on integrated system testing. There was no software test plan worthy of the name and little specification. Unit-level and concurrency testing of the software were essentially absent.

The assumption that was never tested

Three separate things were reused across the transition, and only one of them was code.

The code came across. The assumption underneath the code came across with it, invisibly, because it had never been written down anywhere: that if the software commanded an unsafe configuration, hardware would refuse it. And the confidence came across, because the earlier machines had a good field record.

Then the interlocks were removed. The code did not change, so nothing in it looked wrong, and nothing in a review of it would have looked wrong either. What changed was the environment the code's correctness depended on, and that dependency existed only in the heads of people who had worked on the earlier machines.

This is the Ariane 5 shape appearing in a safety architecture rather than in a numeric range. There, a conversion was safe inside Ariane 4's velocity envelope and the envelope changed. Here, software was safe inside an envelope of hardware guarantees and the guarantees were withdrawn. In both cases a reused module carried a correctness argument scoped to its old environment, and in both cases the argument was left behind while the code moved.

The difference is that a numeric envelope can be written down and compared. An assumption about a guarantee somebody else was providing usually cannot, because nobody ever articulated it as an assumption. It was just how the machine worked.

What testing would and would not have reached

Taking the specific mechanisms one at a time, because they have different answers.

The race condition. A stress test scripting rapid operator edits surfaces it. This is a negative control against the timing path: drive the interface faster than the design assumed and confirm the control task stays consistent. The manufacturer's approach, integrated testing under nominal operation, could not find it, because a nominal operator does not type that fast and a test written by the developer reproduces the developer's pace.

The counter overflow. A test forcing the counter to its boundary surfaces it. Boundary testing on a one byte counter is elementary and it was absent, along with unit testing generally.

The architectural assumption. This is the one that matters and the one no test at either level reaches unless somebody first notices that the assumption exists. The technique, and it is underused, is to write the test that removes a guarantee the design depends on and see whether anything notices. Take the interlock assumption out explicitly and ask what the software does alone. That is a design review question that produces a test, not a test that produces a finding.

The institutional failure. Nothing in a test suite reaches this. Early incident reports were dismissed. Clinics reporting injuries were told the machine could not do what they were describing. A system that produces red and is not believed is the same failure as a test suite nobody trusts, and it is the reason the negative control discipline is necessary and not sufficient.

The honest verdict

Would a negative control have caught it? Partially, and only some of it.

Scripting rapid edits and forcing the counter boundary would have surfaced two specific mechanisms. That is real and it would have mattered. But presenting the Therac-25 as a missing test is a claim the investigation does not support, and repeating it teaches the wrong lesson to exactly the people who most need the right one.

The right lesson has two halves. The first is that when you remove a protection, every piece of software that assumed it becomes unverified, whether or not anybody remembers that it assumed it. The second is that a discipline of proving your checks can fail is worth nothing in an organisation that does not act on red. Knight Capital's 97 ignored warning emails are the same failure in a different industry, on a shorter timescale.

The general argument is in a test that has never failed is a decoration. For the practice of deliberately corrupting inputs and confirming detection, which is the technique this case points at, see the failure mode coverage tool.

FAQ

What went wrong with the Therac-25? Hardware interlocks present on earlier machines were removed and the job given to software that had been written assuming they were there. A race condition reachable by fast operator edits and a one byte counter overflowing to zero combined to bypass a safety check, firing the beam in high energy mode without the spreading target.

Was the Therac-25 caused by a missing unit test? No. Leveson and Turner locate the cause in systemic software-engineering and concurrent-design errors together with institutional failures. Specific tests would have surfaced specific mechanisms, but framing the whole accident as a test gap misrepresents the investigation.

What testing would have surfaced the race condition? A stress test scripting rapid operator edits, driving the interface faster than the design assumed. The manufacturer relied almost exclusively on integrated testing under nominal operation, which cannot reach a timing path that requires an unusually fast operator.

What does Therac-25 teach about reused code? That reused code carries safety assumptions about its old environment, and those assumptions are often unwritten because nobody ever regarded them as assumptions. Removing a hardware protection silently invalidates every piece of software that depended on it.

Why does the institutional half matter? Because early incident reports were dismissed. A system that produces a warning nobody believes has the same practical value as no warning, which makes acting on red the other half of any verification discipline.

Free checklist
The Verification Diligence Checklist

Nine controls, each tied to a public failure that cost somebody nine figures or a life, with the four artifacts a verification data room holds and the one question an investor should ask in the room. Built to be printed and ticked only where somebody has actually watched the check go red.

One email, the resource, and nothing else unless you reply.
Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program