You Can Be 90 Percent in the Green Zone and Measuring Nothing

Mostafa Dhouib·7 min read

You Can Be 90 Percent in the Green Zone and Measuring Nothing

The short answer. Two standard reassurances about a physiological model can both be true while the model has learned nothing. A clinical error grid can show most points in its acceptable zones, and the mean absolute relative difference can look respectable, for a model that simply predicts near the population average every time. The measurement that separates tracking from mean-hugging is the variance explained against a mean baseline, and the split it has to be measured under is subject-disjoint. A published reproduction of five photoplethysmography-based glucose models found that moving to subject-disjoint evaluation collapsed them to near-zero or negative variance explained, with one model's error moving from around 12 percent to around 31 percent on the same architecture and the same data.

A startup has a number. The model estimates something physiological from a signal, the accuracy looks publishable, the error grid is mostly green, and the next step is obviously a bigger model.

Before the bigger model, there is a cheaper question: is the number real? It is answerable in about two weeks, and surprisingly often the answer is no.

Two reassurances that do not reassure

The error grid. Clinical error grids were designed to answer a specific question: would acting on this reading harm the patient? They bin predictions by clinical consequence, which is the right framing for safety and the wrong framing for capability. A prediction sitting near the population average is usually clinically unremarkable, so it lands in an acceptable zone almost by construction. High green-zone percentage is therefore compatible with a model that has learned nothing about the individual in front of it.

The error percentage. Mean absolute relative difference measures typical deviation. For a quantity whose population distribution is reasonably tight, always guessing near the centre produces a modest number. It is a measure of how far off you are on average, not a measure of whether you are responding to the input.

Neither metric is wrong. They answer questions that are not the one you are asking.

The measurement that distinguishes them

Variance explained against a mean baseline, usually reported as R², asks directly: does this model do better than always predicting the average?

A model that tracks the signal scores above zero. A model that predicts the average scores around zero. And a model that moves in response to the input but in the wrong direction scores below zero, which is the result that should end a project in its current form.

Negative variance explained alongside a respectable error percentage and a green error grid is the specific signature of mean-hugging. All three can be true at once, which is why reporting only the first two is so comfortable and so uninformative.

And the split has to be subject-disjoint

The second half is about evaluation design, and it is the half that usually produces the number in the first place.

Physiological data comes in recordings per subject. Split those recordings randomly and the same person appears in both training and test. The model does not have to learn the physiological relationship at all. It can learn the person: their baseline, their signal morphology, their characteristic offset. Then at test time it recognises them and reproduces what it memorised.

That is leakage, and it inflates every metric simultaneously. The error grid improves, the error percentage improves, and the variance explained improves, so nothing looks anomalous.

A subject-disjoint split puts whole people on one side or the other. No subject appears in both. The test is then the condition the product actually operates in, which is a person it has never seen.

A reproduction from a published academic group did exactly this to five existing models. Under subject-disjoint evaluation the variance explained collapsed to near zero or below. One model went from roughly 0.60 to below zero, with its error moving from around 12 percent to around 31 percent. Same architecture, same data, only the evaluation made honest.

Why a bigger model makes this worse

The reflex when accuracy disappoints is capacity. More parameters, a foundation model, more data.

Capacity added on top of a leaking evaluation produces a more confident wrong claim. The extra capacity is better at memorising subjects, which raises the leaked metrics further while doing nothing for a person the model has not seen. You invest a quarter and the number on the slide improves, which is the worst possible feedback.

Establishing that the existing number is real costs about two weeks: re-split by subject, re-run, report variance explained against a mean baseline alongside the error grid. If the number holds, the capacity work is well founded. If it does not, you have saved a quarter and learned where the actual problem is.

What to report instead

Four things together, and the combination is what makes it honest.

The split, stated explicitly, and subject-disjoint by default. Variance explained against a mean baseline, reported next to the error metric rather than instead of it. The error grid, kept, because clinical consequence is a real question. And cohort composition, because an aggregate figure over an unrepresentative sample is a different claim than it appears to be. The relevant consensus standard for wearable heart-rate validation, for instance, requires stratification by skin type, sex and body mass index, which is a reasonable model for anything else of this kind.

Add a fifth if the product allows it: an abstention path. A model that declines when signal quality is poor is more useful than one that always answers, because in physiological measurement the low-quality cases are both common and exactly where confident wrong answers do damage.

The honest limit

The reproduction figures above are from a published academic group working on photoplethysmography-based glucose estimation specifically. They are strong evidence about that problem and about the mechanism, and they are not a general law. A different modality with a genuinely subject-independent relationship may hold up far better under a subject-disjoint split.

The point is not that physiological models do not work. It is that the standard reporting pair, error grid and error percentage, cannot distinguish one that works from one that is averaging, and the measurement that can is cheap to add.

FAQ

Can a model be 90 percent in the clinical error grid's acceptable zones and still be useless? Yes. Error grids bin predictions by clinical consequence, and a prediction near the population average is usually clinically unremarkable, so it lands in an acceptable zone almost by construction. A model that always predicts the average can therefore post a high green-zone percentage while responding to nothing.

What does negative R² mean for a prediction model? It means the model explains less variance than simply predicting the mean every time, so it is actively worse than a constant baseline. Negative variance explained alongside a respectable error percentage and a green error grid is the signature of mean-hugging, and all three can be true at once.

What is a subject-disjoint split and why does it matter? It puts whole people entirely on one side of the train and test boundary, so no subject appears in both. Without it, a model can learn to recognise individuals and reproduce their memorised baseline rather than learning the physiological relationship, which inflates every metric at once so nothing looks anomalous.

Will a bigger model fix poor accuracy on physiological data? Not if the evaluation leaks. Additional capacity is better at memorising subjects, so it raises the leaked metrics while doing nothing for an unseen person, and you get a more confident wrong claim with better-looking numbers. Establish that the existing figure survives a subject-disjoint split first, which takes about two weeks.

What should I report alongside accuracy for a physiological model? The split, stated explicitly and subject-disjoint by default. Variance explained against a mean baseline, next to the error metric rather than instead of it. The error grid, because clinical consequence is a real question. And cohort composition, since an aggregate over an unrepresentative sample is a different claim than it appears.

How much does an honest split actually change results? In one published reproduction of five photoplethysmography-based glucose models, moving to subject-disjoint evaluation collapsed variance explained to near zero or below across all of them, with one going from roughly 0.60 to negative and its error moving from around 12 percent to around 31 percent, on the same architecture and the same data.