A family of physiological estimation systems, built end to end: three sensing modalities, one of them a camera, taken from raw signal through evaluation into deployment and monitoring. The field has a well-known habit. A model is trained, tested on a random split of the recordings, and posts an excellent score. Then it meets someone it has never seen and the score falls apart. By that point the number is usually in a product claim or a regulatory submission, and it has to be withdrawn in public.
A test score is only honest if nothing about the test set leaked into training, and with people that is easy to get wrong.
Every machine learning score rests on one promise: the examples used to grade the model were kept away from it while it learned. While that promise holds, the score is a fair estimate of how the model will do on new data. Once it breaks, the score measures memory. The usual way to keep the promise is to shuffle the dataset and split it at random, most of it for training and the rest for testing.
Physiological data quietly breaks that default. A dataset is normally many recordings taken from a modest number of people, and each person's recordings share a signature: their physiology, how a sensor sits on them, how they move. Split the recordings at random and the same person lands on both sides. The model can then score well by learning to recognise individuals and recall what their values tend to be. That skill is worthless on a stranger, and nothing in the training curve or the test score reveals that it is happening. The number simply comes out higher.
A subject-disjoint split fixes this by keeping all of a person's data together and placing whole people on one side or the other. The test then asks the question a product actually faces: how well does this work on someone it has never met. Two companion checks finish the job. Duplicate and overlap detection catches the same recording entering twice under different names, which is a quieter route to the same leak. Comparison against a model that always predicts the population average catches a model that has learned only the average, which can look acceptable on an error metric while telling you nothing about the individual.
The general principle is that leakage has a direction. It can make a model look better than it is and never worse. That is why it survives review so often. A bad score gets investigated. A good score gets written up, and the investigation that would have found the leak never starts.
The pull, every time, is toward more capacity: a bigger network, a foundation model, one more modality. Capacity only helps if the score it improves is honest. A model that has learned to predict something close to the population average can land inside the acceptable zones of a consensus error grid and post a low average error, while explaining less of the difference between people than the average itself does. Adding capacity on top of an evaluation that leaks produces a bigger, more confident number that is exactly as wrong.
Evaluation bugs look like success, so nobody goes looking for them.
A bug in training usually shows up as a bad score, and bad scores get investigated. A bug in evaluation usually shows up as a good score, and good scores get celebrated, published and built on. In physiological estimation the most common evaluation bug is leakage between subjects, and it is common precisely because the default tooling produces it and the result is pleasant.
The second pattern is a metric that a lazy model can satisfy. Error metrics and clinical error grids reward predictions that land near the truth. In a population where most values cluster in a fairly narrow range, predicting the middle of that range lands near the truth most of the time. Without a comparison against that trivial baseline, a model that has learned nothing about the individual in front of it cannot be told apart from one that has.
The reflex, and the fix.
Report accuracy on a random split
It is the default, it is what the tooling does, and the number it produces is flattering.
A random split puts recordings from the same person on both sides of the line. The model is then graded partly on recognising people it has already met, which is a different skill from doing the task on a stranger.
Prove it on people the model has never seen
Evaluation grouped by subject, so no person appears in both training and test, a validation set kept apart from the test set, and automatic checks for overlapping or duplicated recordings.
A score that holds when the system meets a new person, because that is the only condition it was ever measured under.
Find out whether the number is real before adding capacity. Leakage does not announce itself, and every later decision inherits it.
Reusable signal representations
Self-supervised learning, by masking parts of the signal and by contrasting related and unrelated segments, produces embeddings that transfer between tasks. They are served from one shared inference backbone, with the model and its preprocessing versioned together and small task-specific heads on top.
Fusing the two contact modalities
The two contact signals are aligned in time and combined through cross-attention, with the physical timing relationship between them used as a coupling. When one channel drops out, a gate notices and the system falls back to what the remaining channel can support.
Leakage-safe validation, the lead facet
Every evaluation is grouped by subject and checked for overlap and duplicates. Scores are calibrated and benchmarks are reproducible from a pinned state. The results feed promotion gates, so no model reaches production on a number that was never real.
A model of signal quality
A separate model learns what artefacts look like, trained with weak supervision and calibrated so its confidence means something. At runtime it gates the estimator. When quality is poor the system defers and asks for the signal to be reacquired, and it gives no answer from noise.
Fusing the camera channel, the harder facet
The contactless channel needs a full computer-vision front end that checks each frame is usable, tracks the subject and stabilises the region being measured. It is fused with the wearable signal through real-time selection of whichever modality is trustworthy in the current operating mode, and validated with whole cameras held out so the score covers devices the model has not seen.
What each choice cost, and what it bought.
AMaking subject-disjoint evaluation the default
Grouped evaluation was made the only way the platform scores a model, with no switch to turn it off. It has a cost. Test sets hold fewer independent units, scores come out lower and noisier, and they are harder to compare with published work that used random splits. What it buys is a number that does not have to be withdrawn. The alternative is a higher figure that fails the first time the product meets a stranger, and that failure lands on whoever repeated the figure.
BWiring validation into promotion
The evaluation results are not a report someone reads. They are a gate a model has to pass before it reaches production, and a model whose evidence was produced under a weaker protocol cannot be promoted at all. That adds friction at exactly the moment teams are in a hurry. It also means a deadline cannot quietly lower the standard, which is the moment the standard matters most.
CTeaching the system to defer
A separate model judges signal quality, and when quality is poor the estimator does not answer. Some readings therefore return nothing, and coverage drops. The trade is worth it because an answer computed from noise looks exactly like a real answer to whoever receives it. A deferral is visible and can be acted on by reacquiring the signal. A wrong value is invisible.
DSelecting modalities by trust
With three modalities available, the tempting design blends all of them all the time. Here each modality is gated, and the system selects whichever it can trust in the current conditions, with explicit support for a channel going missing. It is more complex to build and to validate. In return a degraded channel cannot drag a good one down, and the camera channel is validated with whole cameras held out, so its score covers hardware the model was never trained on.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
No self-reported accuracy figure is claimed, deliberately. Validation method is the claim.
A strong number on a random split, a product claim built on it, and a collapse the first time the system meets people it has never seen, discovered after the claim is public.
Physiological estimation systems that report accuracy under the split that matters, keep working when a modality goes missing, and defer when the signal is too poor to read. One shared backbone and reusable heads carry the work across tasks, and promotion gates stop any model whose evidence was produced under a weaker protocol.
What it owns, and what it hands to a person.
No accuracy figure appears on this page, and that is deliberate. The validation method is the claim. A figure produced under a weaker split would be larger and would mean less. A figure produced under the right split still only describes the subjects, devices and conditions it was measured on, so domains outside the validated set are flagged and no claim is made for them.
Combining the most information does not win. The strongest system knows which of its inputs to trust under current conditions and degrades gracefully when it can trust none of them. The same discipline carries to any estimation problem where the data has structure per person, per site or per device, which is most of them.
A random split of recordings lets the same person appear in training and test, so the model can score well by recognising people instead of doing the task.
Leakage only ever makes a score look better, which is why it survives review. Rule it out by design, because nobody investigates a result that came out well.
Always compare against a model that predicts the population average. A low error and a pass on a clinical error grid can both come from a model that has learned nothing about the individual.
A system that defers when its input is poor is more useful than one that always answers, because a deferral is visible and a wrong value is not.
Before adding capacity, confirm the number you are trying to improve is real. Capacity added on top of a leaking evaluation produces a more confident wrong claim.