List what can go wrong, how often, and what actually notices. Frequency can be a guess; the ranking is far less sensitive to it than to whether anything detects the failure at all. Start with what has happened, then add what somebody has worried about out loud.
| Failure | Per year | Consequence | Detected by | Detect in min | Safe state | Remove |
|---|---|---|---|---|---|---|
| Failure | Per year | Consequence | Detected by | Detect in | Safe state | Coverage |
|---|---|---|---|---|---|---|
| Upstream dependency times out | 800 | Degraded | An alert | 2.0 min | Same layer | 52% |
| Model returns a partial result as complete | 300 | Degraded | Nothing | never | None | 0% |
| Retry after partial success double-executes | 6 | Unsafe | A metric | 24 h | Same layer | 8% |
| Tool call hangs indefinitely | 12 | Outage | A person | 45 min | None | 3% |
| Deployed artifact differs from evaluated | 2 | Outage | Nothing | never | None | 0% |
| Nested call tree exceeds spend budget | 24 | Contained | An alert | 15 min | Enforced below | 61% |
Detection and safe state, by failure class
Ten failure classes that recur across production systems, with what actually detects each and what the safe state should be. The right-hand column is the pattern worth internalising: in most of these the obvious instrument is structurally unable to see the failure.
| Failure class | What detects it | Safe state | Why the obvious instrument misses it |
|---|---|---|---|
| Hangs indefinitely | Wall clock outside the loop | Abandon and report gave-up | The counter freezes; only a clock catches it |
| Spins without progress | Remaining-work measure stalling | Abandon at the stall threshold | The clock catches it only at full cost |
| Returns a partial result as complete | Type-level distinction at the caller | Refuse rather than return | Silent by construction; nothing alerts |
| Silently falls back to a default | A counter on the fallback path | Mark the value as degraded | Returns HTTP 200; the dashboard stays green |
| Double-executes after a retry | Reconciliation against an outside record | Idempotency key at the effect | Prevention fails silently; only comparison sees it |
| Deployed artifact differs from evaluated | Hash comparison in the deploy path | Refuse to start | Version strings lie; compare bytes |
| Input distribution has moved | Drift monitor on the inputs | Refuse out-of-distribution inputs | Accuracy metrics cannot see it without labels |
| A node goes silent | Time since last successful contact | Mark unknown, never assume healthy | Absence is not a signal; it must be constructed |
| Dependency degrades but responds | Latency percentile, not error rate | Circuit break to the deterministic path | Error-rate alerts never fire |
| Correlated failure across a shared resource | Blast-radius grouping in alerts | Per-tenant or per-zone limits | Looks like many incidents, is one |
Every row in the last column is a case where a reasonable-looking monitor reports health while the failure is occurring. That is why coverage has to be scored on what the detector can see rather than on whether a detector exists.
A worked example
Six failure modes on an agent platform, which is roughly what a team produces in twenty minutes when asked what can go wrong. Two have alerts, one has a metric somebody reads, and three have nothing.
Risk-weighted coverage comes out at 35 percent, with two of six failures detected by nothing and three of six having no safe state at all. But the useful part is the ordering, and it is not what anyone expects.
The largest exposure is upstream dependency timeouts, at 800 a year. That row is already 52 percent covered: there is an alert, it fires in two minutes, and there is a defined fallback. It tops the list purely on frequency. Acting on it would be the obvious move and close to the least useful one, because most of what can be recovered there already has been.
Second is the model returning a partial result as complete, three hundred times a year at zero percent coverage. Nothing detects it and there is nowhere safe to go. That is where the cheapest improvement lives: an alert on that path alone takes overall coverage from 35 to 42 percent. Not a rewrite, not a safe state, just a counter and a threshold on a path that currently returns success.
The third finding is in the safe-state column and it is the one a spreadsheet hides. The double-execution row has a safe state defined in the same layer that fails, which reads as covered on any risk register. It is not: if the component that is wrong is the component that decides to enter the safe state, there is no safe state. The tool flags those separately from the ones with nothing at all, because they look identical in a list and behave completely differently.
The arithmetic, so you can check it
Risk per failure is frequency x consequence weight, where the weights are 1 for contained, 3 for degraded, 8 for an outage and 25 for unsafe. Coverage per failure is detection x latency discount x (0.35 + 0.65 x safe state). The floor of 0.35 exists because knowing about a failure is worth something even with nowhere safe to go.
The latency discount is 1 / (1 + log2(1 + minutes / 60)), floored at 0.25, so a detector that takes an hour keeps most of its credit and one that takes a week keeps a quarter of it. Overall coverage is exposure-weighted rather than a simple average, since a well-covered rare failure should not offset an uncovered frequent one.
The weights are a judgement, not a measurement, and yours may differ. What the tool is really doing is forcing three questions to be answered explicitly per failure, and the ranking is far more robust than the absolute coverage number. Treat the percentage as a way to compare before and after a change rather than as a score to report. It also assumes failures are independent, which correlated infrastructure failures are not.
The reasoning is in your green dashboard is not evidence and human in the loop or safe state. For sizing the fuses that back these up, use the fuse sizing tool.
Questions
A list of what can fail, how often, what it costs, what detects it, how long detection takes, and what safe state follows. Full FMEA is heavyweight enough that software teams skip it entirely, which means the failures nothing watches for stay undiscovered until one of them happens. Six rows and twenty minutes finds most of what a formal process would.
Because a failure found by a customer complaint a week later is barely detected. The exposure that matters is what accumulates between the failure occurring and somebody knowing, so a detector that takes a day earns a fraction of the credit an in-line check earns. This tool discounts detection by latency for that reason.
That the safe action is applied by something the failing component cannot influence. If the component that is wrong is also the component that decides to enter the safe state, there is no safe state, only an intention. That is why an envelope limit in a controller counts and a check inside the code that might be broken does not.
Much less than people expect. The ranking is dominated by whether anything detects the failure and whether a safe state exists, and both of those are yes-or-no. An order of magnitude on frequency rarely changes which failure sits at the top of the list, so guess and move on rather than stalling on the number.
Add detection to the highest-exposure failure that nothing watches. Detection is nearly always cheaper than a safe state, and it converts a silent failure into one somebody can act on, which is the larger part of the improvement. The tool computes what that single change does to coverage so the trade is visible before you spend anything.