Every Risk Control Is Verified Twice

Mostafa Dhouib·10 min read

Every Risk Control Is Verified Twice

The short answer. ISO 14971, the international standard for risk management of medical devices, requires two separate verifications for each risk control measure. One that the control was implemented, and one that the control is effective at reducing the risk it was introduced to reduce. These are different questions with different evidence. Implementation asks whether the thing exists and does what it says. Effectiveness asks whether the hazardous situation it was meant to prevent is actually less likely or less harmful now. Most software risk management plans I review write only the first and call it done. The second is harder, because it requires you to reproduce the hazard and watch the control stop it.

I review a lot of risk management files. The traceability is usually present, the controls are usually sensible, and the verification column usually points at a unit test. That unit test is nearly always evidence of implementation only.

What the standard asks for, and why it asks twice

The structure of the standard is a loop. Identify hazards. Estimate risk. Decide whether it needs reducing. Select and implement risk control measures. Verify. Evaluate the residual risk. Check that controls have not introduced new hazards. Then review the whole thing.

Inside the verification step there are two distinct obligations, and the standard separates them deliberately.

Verification of implementation. The risk control measure that the file says exists, exists. It is in the product, it is in the released configuration, and it does the thing the control was specified to do.

Verification of effectiveness. The implemented measure actually reduces the risk. The hazardous situation is less probable, or the harm is less severe, or the sequence of events leading to harm is interrupted.

The reason for two obligations is that the first does not imply the second. A control can be present, correct against its own specification, and useless against the hazard. That happens when the hazard was misunderstood, when the control sits in the wrong place in the sequence of events, when it covers one path to the hazard and not the others, or when a human in the loop does not respond to it the way the file assumes.

It is the same distinction as the difference between a test existing and a test having been observed to fail, which is the subject of a test that has never failed is a decoration. Implementation verification tells you the control is there. Effectiveness verification is the part where you try to break through it and fail.

The distinction in software terms

Abstract statements of this never land. Examples do.

An input range check on a dose or rate field. Implementation verification: the field rejects values outside the specified range, and a test covers the boundaries. Effectiveness verification: no path exists by which an out-of-range value reaches the actuation logic. That means checking the API that bypasses the form, the import path, the stored configuration, the default used when the field is absent, the unit conversion that happens after validation, and the retry that replays an old payload. The check being correct and the hazard being prevented are not the same claim.

A watchdog that forces a safe state when a task stops responding. Implementation verification: when the task stops calling the watchdog, the watchdog fires. Effectiveness verification: inject the realistic failure. The task does not die, it blocks on a lock. Or it dies but a supervisor restarts it into a partially initialised state. Or it keeps servicing the watchdog from a timer interrupt while the control loop itself is wedged, which is the classic way a watchdog verifies as implemented and fails as effective.

An alarm for an out-of-range physiological reading. Implementation verification: the threshold triggers, the alarm renders, the sound plays. Effectiveness verification: does the alarm interrupt the hazard sequence in use? If the clinical environment generates many such alarms, if the alarm is silenced by default on that screen, if the reading itself can be stale and still look valid, or if the sensor failure mode produces a plausible in-range value rather than an out-of-range one, then the control is implemented and the hazard is unmitigated.

A permission boundary preventing an unauthorised user from changing a treatment parameter. Implementation verification: the unauthorised role gets a rejection. Effectiveness verification: enumerate every path to that parameter. A background job, a support tool, a bulk import, a cached token, a different service calling internally without the check. Boundaries fail at the paths nobody listed, not at the path the test exercises.

A model output gated by a confidence threshold. Implementation verification: outputs below the threshold are suppressed. Effectiveness verification: are the wrong outputs the low-confidence ones? If the failure mode is confident and wrong, the threshold is implemented, measurable and ineffective. This is why confidence is not evidence, and it is one of the clearest cases where only an effectiveness test finds the problem.

A checksum on a stored record to detect corruption. Implementation verification: corrupt a byte, see the checksum fail. Effectiveness verification: is the checksum computed and stored in a way that survives the failure you are protecting against? If it is written in the same operation as the data, a partial write may produce a consistent pair of wrong values. If it is recomputed on read from the data just read, it detects nothing.

The pattern across all six is the same. Implementation verification tests the control against its own specification. Effectiveness verification tests the control against the hazard, which means you have to produce the hazard.

Why effectiveness is the harder test

It needs the hazard reproduced, not simulated loosely. You must create the conditions in which harm would occur and demonstrate the control interrupts the sequence. For software that often means fault injection, which is its own discipline, and for anything involving a person it may mean usability work rather than a test.

It depends on the correctness of the hazard analysis. If the sequence of events in the file is wrong, an effectiveness test built from that sequence verifies the wrong thing convincingly. This is the failure that is hardest to catch from inside, because the evidence looks complete.

Multiple paths. A hazard usually has more than one route. Implementation verification covers the control. Effectiveness verification has to cover the hazard, which means enumerating routes, and enumeration is the part that gets truncated when the deadline approaches.

Human response is part of the control. Where the measure is information for safety, an alarm or a label or a warning, effectiveness depends on a person noticing and acting. That is not establishable by code review. It needs evaluation with representative users in representative conditions, and the result is sometimes that a technically perfect alarm does not work.

Interaction effects. A control can be effective in isolation and lose effectiveness in combination with another, or introduce a new hazard. The standard requires checking for new hazards introduced by controls, and that check also belongs to the effectiveness side of the work.

How this maps onto ordinary engineering verification

The good news is you probably already do most of the first half and some of the second under different names.

Unit and integration tests against a control's specification are implementation verification. Keep them. Trace them. Do not let them stand in for the other column.

Fault injection, negative testing and chaos work are effectiveness verification. Removing a sensor, corrupting a message, killing a process, saturating a queue, and confirming the system reaches a safe state is exactly what the standard is asking for, and it is usually already happening somewhere in the engineering process without being traced into the risk file.

Property-based and fuzz testing contribute to path enumeration. They attack the assumption that the paths you thought of are the paths that exist, which is the main weakness of effectiveness verification.

Mutation testing provides evidence that a test can fail. A control whose verification test passes on a deliberately broken implementation is not verified at all, in either column.

Usability evaluation is verification of effectiveness for information-based controls. It belongs in the risk file, not only in the design file.

The practical change is mostly bookkeeping plus honesty. Add a second verification column to the risk control table. For every control, name the evidence that it was implemented and, separately, the evidence that it is effective, with the method and the record. Then look down the effectiveness column. The rows where the same unit test appears in both columns are the rows where half the work is missing, and they are the rows to start on.

The honest limit

I am describing the structure of the requirement, not quoting clause text. The standard is paywalled, the clause numbering and wording differ between editions and national adoptions, and your notified body or reviewer will care about the edition that applies to you. Treat the two-verification structure as the reliable part and confirm the exact clause references, wording and any amendment or guidance that applies against the current text before you build a submission around it.

Effectiveness verification is also not always fully achievable. Some hazards cannot be reproduced ethically or physically, and the honest response is a documented rationale combining analysis, bench reproduction of a surrogate condition, and literature or field data, rather than a test you pretend is equivalent. A reviewer will generally accept a well-argued limitation. What they will not accept, and what I see more often, is an effectiveness column quietly filled with implementation evidence.

Finally, none of this makes the residual risk evaluation easier. Verifying a control is effective tells you it works. It does not tell you the remaining risk is acceptable, and that judgement sits outside the verification activity entirely.

FAQ

What are the two verifications ISO 14971 requires for a risk control? Verification that the risk control measure was implemented, and separate verification that it is effective at reducing the risk. The first asks whether the control exists in the released product and meets its own specification. The second asks whether the hazardous situation it targets is genuinely less likely or less harmful now that the control is in place.

Why is implementation verification not enough? Because a control can be present, correct against its specification, and still fail to reduce the risk. The hazard may have been misunderstood, the control may sit at the wrong point in the sequence of events, it may cover one route to the hazard and not others, or a person may not respond to it the way the risk file assumes. Only the effectiveness test exposes those.

Can you give a software example of the difference? A watchdog that forces a safe state. Implementation verification shows that when the task stops servicing the watchdog, the watchdog fires. Effectiveness verification injects the realistic failure, where the task blocks on a lock rather than dying, or keeps servicing the watchdog from a timer while the control loop is wedged. The control verifies as implemented and fails as effective.

Why is effectiveness harder to verify? It requires reproducing the hazard rather than testing the control, so it depends on fault injection, on an accurate hazard analysis, and on enumerating every route to the hazard rather than the one route the test exercises. For controls that are information for safety, such as an alarm or a warning, it also depends on how real users respond, which no code review can establish.

Does existing engineering testing count towards either verification? Yes, and most teams already generate both kinds without tracing them. Unit and integration tests against a control's specification are implementation evidence. Fault injection, negative testing, failure-mode work and usability evaluation are effectiveness evidence. The change needed is usually to separate the two columns in the risk control table and name the record for each.

How do I find the gaps in an existing risk file quickly? Add a second verification column and fill both from the records you have. Any row where the same test appears as evidence for implementation and for effectiveness has only half its verification done. Those rows, ranked by the severity of the hazard they address, are the work queue.