Fresh, Entitled And Real Are Three Properties With Three Mechanisms

Mostafa Dhouib·10 min read

Fresh, Entitled And Real Are Three Properties With Three Mechanisms

The short answer. When a figure reaches somebody who will act on it and answer for it, three separate things have to be true, and each one needs its own mechanism. Fresh, meaning the feed behind it has not gone quiet, which requires an alarm that fires on absence rather than on a bad value. Entitled, meaning this particular person is permitted to see this particular figure, which requires a check at the moment of presentation. And real, meaning it traces to a source of record, which requires an abstain path for when it does not. Conflate them and you build the easiest one, pass your own review, and fail the other two without a signal.

An adviser is looking at a screen. The figure on it will go into advice given to a client, and the adviser's name will be on that advice.

Everyone involved agrees the number must be correct. That agreement is where the trouble starts, because "correct" is not one property. It is three, they fail independently, and they are satisfied by three unrelated mechanisms.

Why conflating them is the actual failure

Teams do not skip these properties deliberately. They build one mechanism, usually the most tractable one, and let it stand in for the others.

The common version is a validation layer. Figures are range-checked, type-checked, perhaps cross-checked against a tolerance. Somebody calls this data quality, and in review it reads as thorough. It addresses none of the three properties completely.

A stale figure passes every range check, because it was valid when it arrived. An unentitled figure passes every range check, because entitlement is not a property of the value. And a fabricated or defaulted figure passes range checks most easily of all, because a plausible default was chosen specifically to look reasonable.

So the system reports healthy. The dashboards are green. What is actually happening is that one of three failure modes is covered and the other two have no detector at all, which is worse than knowing they are uncovered, because the green dashboard is read as evidence about all three.

The rest of this is the three mechanisms, kept separate on purpose.

Fresh needs an alarm on absence

A feed that stops does not emit a wrong value. It emits nothing.

Threshold alarms watch values. Nothing arrives, so no threshold is crossed, so nothing fires. Meanwhile the serving layer continues to return whatever it has, because the last good value is still in the cache or the table, and it is returned with no indication of when it was obtained. The adviser sees a price that looks exactly like a current price.

The mechanism is a separate alarm lane keyed on arrival rather than on content. For each feed, state the expected cadence. If nothing has arrived within the window, raise an alarm. The alarm condition is silence.

Two design requirements make it work.

Timestamp the observation, not the fetch. Each value carries the time the source produced it, not the time you read it. A cache refreshed from a stale upstream looks fresh if you record your own read time, and this is a surprisingly common way freshness monitoring deceives itself.

Carry the age to the point of presentation. Freshness is not only an operations concern. The screen that shows the figure should know how old it is and should be able to mark it, or withhold it, when it is beyond tolerance. An alarm that pages an engineer while the adviser continues to see the stale number has protected the system and not the decision.

Entitled needs a check at presentation

Entitlement is a statement about a person and a figure together, so it cannot be determined anywhere upstream of the pair being brought together.

Market data carries licence conditions. Some figures may be seen by a professional user and not a retail one, some only in delayed form, some not redistributed at all. Internally, a client's holdings may be visible to the adviser who owns the relationship and not to the whole firm. Both kinds of restriction attach to the combination of viewer and value.

The mechanism is a check at the moment of presentation, against the object being requested, derived from the viewer's authenticated identity.

Where this goes wrong:

Checking at ingest. A pipeline that filters on entitlement when data arrives has made one decision for every future viewer. The next consumer inherits a filter chosen for somebody else, and it will be either too permissive or too restrictive. Neither is detectable from the data.

Checking the screen instead of the figure. Gating access to a page and then returning whichever record the request names is the familiar object level authorisation failure. The page check passes, and an altered identifier returns a figure the viewer is not entitled to.

Treating entitlement as static. Licence tiers, client relationships and staff roles all change. A check evaluated at session start, or cached for the day, serves yesterday's permissions. The evaluation belongs on the request.

Forgetting the derived figure. A number computed from entitled inputs inherits their restrictions. An aggregate, a chart or an export built from restricted data is restricted, and export paths are where this is most often missed, because the entitlement check lives in the view code the export bypasses.

Real needs a trace and an abstain path

The third property is the one that gets argued about: the figure has to correspond to something a source of record actually said.

The failures here are not corruption. They are substitution. A default supplied when a lookup returned nothing. A value interpolated across a gap because the chart looked wrong with a hole in it. A figure carried forward from the previous period because the current one had not loaded. A model output presented in the same visual position as a measured value. Each of those is a number that no source of record ever asserted, presented as though one had.

The mechanism has two halves and both are required.

Provenance. Each figure retains a pointer to what produced it: which source, which identifier, which observation. Not a free-text note about the pipeline, a reference you can follow to the original assertion. Provenance is what makes the property testable rather than assumed.

An abstain path. When the trace is missing or broken, the correct output is not a number. It is a refusal to produce one, surfaced as such. This is the part teams resist, because a blank is visibly incomplete and a plausible default is not. That is exactly the argument for it. An interface that cannot show absence will be filled with substitutes by the engineers who have to make it render.

Abstaining needs to be designed, not bolted on. Every component in the path needs a representation for "no value and here is why", and every consumer, including charts, exports and downstream calculations, has to handle it. Retrofitting absence into a system whose types do not permit it is how defaults get introduced in the first place.

Keeping them separate in practice

The point of separating the three is that each gets its own assertion, and each can fail visibly.

Carry three attributes alongside every figure: the source observation time, the entitlement context it was resolved under, and the provenance reference. Then each property has somewhere to be checked and somewhere to be tested.

Test them independently too, because a single test will quietly cover one. Stop a feed and assert that an alarm fires and the figure is marked stale. Request a figure as a viewer without entitlement and assert refusal, including through the export path. Break a provenance link and assert that the system abstains rather than substituting. Three tests, three mechanisms, three different kinds of failure.

The honest reason to be strict about this is that all three failures are silent. A stale number, an unentitled number and a fabricated number all look exactly like a correct number on the screen. None of them produces an error. The only thing distinguishing them is a mechanism designed to notice, which is why the absence of any one of the three cannot be inferred from the system appearing to work.

The honest limit

These three properties are necessary and not sufficient. A figure can be fresh, entitled and traceable to a source of record and still be wrong, because the source was wrong. Provenance tells you who asserted something, not whether the assertion was right. Detecting that needs independent corroboration, which is a fourth concern and a harder one.

Freshness windows are also a judgement rather than a derivation. Too tight and the alarm fires on normal variance until people mute it, which is worse than no alarm because it looks covered. Too loose and a feed can be quiet for hours inside tolerance. The window has to come from the decision the figure supports, and it is usually wrong on the first attempt.

Entitlement at presentation has a cost. It means a check per figure per viewer, on a path that is often latency sensitive, and the temptation to cache the decision is strong. Cache the inputs to the decision if you must, with a short expiry, rather than the outcome.

And abstaining has a real failure mode of its own. A system that refuses too readily trains its users to work around it, and a spreadsheet maintained beside the platform is a worse outcome than a marked estimate shown inside it. Abstention needs to be rare enough to be taken seriously, which means the engineering work is in reducing the cases that trigger it, not in widening the tolerance that hides them.

FAQ

Why aren't freshness, entitlement and provenance one data quality problem? Because they fail independently and no single mechanism detects all three. A stale value passes every range check, since it was valid when it arrived. An unentitled value passes too, because entitlement is a property of the viewer rather than the number. A fabricated default passes most easily of all, having been chosen to look plausible. One validation layer covers none of them fully.

What kind of alarm catches a feed that has gone quiet? One that fires on absence. For each feed, state the expected cadence and raise an alarm when nothing has arrived inside the window. Threshold alarms watch values, and a stopped feed sends no value, so there is nothing for them to trigger on while the last good number keeps being served as though it were current.

Why must entitlement be checked at presentation rather than at ingest? Because entitlement is a statement about a viewer and a figure together, so it cannot be resolved before the two meet. Filtering at ingest fixes one decision for every future consumer, and that decision will be wrong for some of them in a way the data cannot reveal. It also has to be re-evaluated per request, since licence tiers and relationships change.

What does it mean for a figure to be real? That it traces back to something a source of record actually asserted, with a reference you can follow to that observation. The failures are substitutions rather than corruption: a default supplied when a lookup found nothing, a value interpolated across a gap, a figure carried forward from last period, a model output occupying the position of a measurement.

What should the system do when the provenance trace is missing? Abstain, and say so. The correct output is no number with a stated reason, not a plausible substitute. That requires every component to have a representation for absence and every consumer, including charts, exports and downstream calculations, to handle it. Systems whose types cannot express absence are the ones that end up full of defaults.

How do I know whether my system actually has all three? Test each separately, because one test will only cover one property. Stop a feed and assert an alarm fires and the figure is marked stale. Request a figure as a viewer without entitlement, including via export, and assert refusal. Break a provenance link and assert the system abstains. All three failures look identical on screen without a mechanism built to notice.