Page On Silence, Not Only On The Value

Mostafa Dhouib·10 min read

Page On Silence, Not Only On The Value

The short answer. Every threshold alarm is a statement about a value that arrived. When a feed stops, no value arrives, so no threshold is crossed and nothing fires. Worse, the serving layer keeps returning the last number it has, with no indication of its age, so the figure on screen looks exactly like a current one. The only mechanism that catches this is a second alarm lane whose trigger is absence: for each feed, an expected cadence, and an alarm when nothing has arrived inside that window. It is the alarm most often missing, because every other one is written against something arriving.

A data feed supplies the figures an adviser works from. Monitoring is in place. Values outside expected bounds raise an alert, error rates are watched, the integration has retries and logging.

The feed stops at eleven in the morning. Nothing alerts. The screens carry on showing the eleven o'clock numbers all day, and advice is given on them.

Everything in that setup was watching for the wrong shape of failure.

Absence is not an out-of-range value

Threshold monitoring has an implicit precondition: there is something to evaluate. A rule saying "alert if the price moves more than five per cent" is a function of an arriving value. With no arriving value, the function is never called.

The failure is quieter than that, though. Three things compound.

No error is generated. A feed that stops often produces no exception anywhere. A scheduled job that completed successfully with zero rows. An upstream that returns an empty payload with a success status. A message consumer with nothing on the queue. Each of these is indistinguishable, from the inside, from a period in which genuinely nothing happened.

The last value keeps serving. Whatever is in the cache or the table is returned to every reader. It is not marked. It does not come with its age. If the serving layer does not carry the source timestamp through to the point of display, there is nothing for a consumer to notice.

Health checks keep passing. The service is up, responding and returning data. Every signal available to an operator says the system is working, and in a narrow sense it is. It is serving. It just has nothing new to serve.

So the silence is invisible from three directions at once, which is why it gets discovered by a person noticing that a number has not moved, often hours later and often in front of a client.

The dead man's switch

The mechanism is simple to describe and the discipline is in the details.

For each feed, declare the cadence you expect. Record the time of the most recent observation. Run an independent check that compares that time against now, and alarms when the gap exceeds the window. The alarm condition is that nothing has happened.

The name comes from control systems, where a device must receive a periodic signal to keep operating and stops if the signal ceases. The logic is inverted relative to ordinary alarms. The normal state requires continuous positive evidence, and the absence of evidence is the fault.

What makes it work in practice:

Use the source timestamp, not your own. The observation time should be when the source produced the value, not when you fetched or wrote it. A pipeline that refreshes from a stale upstream and stamps its own clock reports perfect freshness while serving yesterday's figures. This single mistake defeats the entire mechanism and it is easy to make, because your own clock is always available and the source's sometimes is not.

Put the check outside the thing it watches. An alarm evaluated by the same process, scheduler or cluster that feeds the data will also be down when that component is down, and a silent monitor watching a silent feed produces no alert. Separate infrastructure, or at minimum a separate scheduler with its own independent heartbeat.

Check at the right granularity. A single aggregate freshness check across a feed hides partial failure. If one instrument, one region or one partition stops while the rest continue, the overall latest timestamp stays current. The check needs to run per logical stream at whatever granularity someone would act on, which usually means more checks than anyone initially wants.

Alarm on the derived figure too, not just the feed. A calculated figure can go stale because one of several inputs stopped, while the calculation itself continues to run happily on whatever it last had. The staleness of an output is the oldest of its inputs, and that has to be computed and carried rather than assumed.

Surface age at the point of use. Paging an engineer is necessary and insufficient. The screen should know the age of what it is showing and mark or withhold it beyond tolerance. Otherwise the alarm protects the system while the person making a decision continues to read a stale number.

Choosing the window

The window is where these alarms are won or lost. Too tight and it fires on normal variation, people mute it, and you now have a muted alarm that reads as coverage. Too loose and a feed can be dead for most of a working day inside tolerance.

Some ways to choose it honestly:

Start from the decision, not from the feed. Ask how old a figure can be before it would change what someone does. If an adviser would act differently on a price twenty minutes old, the window is well under twenty minutes, regardless of how often the source publishes. The tolerance belongs to the use, not the pipeline.

Measure the observed inter-arrival gap before setting it. Collect the actual distribution of gaps over a representative period and set the window above the normal tail with headroom. Setting it from the documented publication frequency produces an alarm that fires on routine jitter, because documented frequency is a nominal figure and real feeds are bursty.

Make the window data-driven, not uniform. Different feeds have different natural cadences, and one global staleness threshold will be wrong for nearly all of them. The window is a property of each stream, declared alongside it, and ideally visible in the same place as the stream's definition so the two cannot drift apart.

Encode the calendar. Many feeds legitimately go quiet overnight, at weekends, on public holidays and outside market hours. A naive window pages someone every Saturday, which is the fastest way to have the alarm disabled. The expected cadence has to include when the feed is expected to be silent, and that calendar is a piece of configuration someone has to own and maintain.

Separate warn from page. A two-level window is usually right. A warning at the point where the gap is abnormal, a page at the point where the data has become unusable for its purpose. That gives an operator the chance to act before the figures on screen are actively misleading.

Alarm on recovery too. When a feed resumes, something should state that the gap existed and how long it was, because figures derived during it may need review. A silence that ends without a record leaves no trace that the period was untrustworthy.

Why this is the alarm most often missing

It is worth being clear about the reason, because it is structural rather than careless.

Every monitoring system makes it easy to write rules about things that happen. You point at a metric, pick a comparison, set a threshold. The entire interface is built around events and values. Writing a rule about something that did not happen requires a different construction, and in most tooling it is possible but not obvious, so it is not the path of least resistance.

Alarms also get written in response to incidents, and the incidents that generate alarms are the ones that were noticed. A loud failure produces an alert within days. A silent failure takes weeks to attribute and is often filed as a data issue rather than a monitoring gap, so no alarm comes out of it.

And absence alarms require a declaration that nothing else requires: a statement of what normal cadence is. That is a per-feed judgement someone has to make and own, which is more work than pointing at an existing metric. Work that has to be done per stream, by someone with domain knowledge, tends to be deferred.

The practical consequence is that most monitoring setups are complete with respect to bad values and empty with respect to no values. If you want one audit to run against your own system, take the list of feeds and ask, for each one, which alarm fires if it stops. Where the answer is a person noticing, that is the gap.

The honest limit

An absence alarm proves data is arriving. It says nothing about whether the arriving data is right. A feed that publishes on cadence with wrong values, or republishes the same unchanged value forever, satisfies every freshness check perfectly. Detecting an unchanging value is a separate check, and distinguishing a genuinely static figure from a stuck one needs domain knowledge that monitoring cannot supply.

Windows will be wrong initially and need revision. Expect a period of tuning, and treat every mute as a defect to be fixed rather than a workaround, because a muted absence alarm is strictly worse than none. It shows as configured coverage.

Calendars are an ongoing operational cost, not a one-off. Market holidays change, sources change their schedules, and a calendar nobody maintains will either page on a quiet day or stay silent through a real outage on a day it thinks is a holiday. That second failure is the one to watch for, because it is itself silent.

And the alarm only helps if someone can act on it. An absence alarm routed to a channel nobody watches, or without a runbook saying who to contact upstream and what to do with figures already served during the gap, has converted a silent failure into an unread notification. The mechanism is the alarm plus the response, and the response is the part that tends to be missing second.

FAQ

Why doesn't a threshold alarm catch a feed that has stopped? Because a threshold is a rule about an arriving value, and a stopped feed sends nothing. The rule is never evaluated, so it never fires. Meanwhile the serving layer continues returning the last number it holds, unmarked and without its age, so the figure looks exactly like a current one to anyone reading the screen.

What is a dead man's switch alarm? An alarm whose fault condition is absence. You declare the cadence a feed is expected to maintain, record the time of the latest observation, and raise an alarm when the gap between that time and now exceeds a window. The normal state requires continuous positive evidence, which inverts the logic of every alarm written against values.

How should I choose the cadence window? Start from the decision the figure supports, by asking how old it can be before someone would act differently. Then measure the actual distribution of inter-arrival gaps and set the window above the normal tail with headroom, rather than from the documented publication frequency, which is nominal while real feeds are bursty. Set it per stream, never globally.

Where should the absence check run? Outside the system it is watching. A check evaluated by the same process, scheduler or cluster that delivers the data goes down with it, and a silent monitor watching a silent feed produces no alert at all. Use separate infrastructure, or at minimum a separate scheduler with its own independent heartbeat.

How do I stop the alarm firing every weekend? Encode the calendar into the expected cadence, including overnight periods, weekends, public holidays and market hours. A naive window pages someone on every legitimate quiet period, which gets the alarm muted, and a muted absence alarm is worse than none because it appears to be coverage. The calendar needs an owner, because schedules and holidays change.

Why is this the most commonly missing alarm? Because monitoring tools make rules about arriving values the path of least resistance, and a rule about something that did not happen takes a different construction. Alarms also get written after noticed incidents, and silent failures are usually filed as data issues rather than monitoring gaps. Absence alarms additionally need a declared normal cadence per feed, which someone has to own.