How to Build a Test Set That Actually Predicts Production

Mostafa Dhouib·September 4, 2026·8 min read

How to Build a Test Set That Actually Predicts Production

The short answer. A random split measures how much your test set resembles your training set. It cannot detect the failure that actually kills systems in production, which is a shortcut learned from a source that appears on both sides of the split. Building a test set that predicts production means holding out whole groups rather than random rows, picking the grouping variable that matches how the system will be deployed, sizing the holdout so the number means something, and checking for leakage before you believe any of it.

Most test sets answer a question nobody asked.

They are built by taking the dataset, shuffling it, and slicing off twenty percent. That produces a number, the number is usually high, and everyone treats it as a prediction about production. It is not one. It is a measurement of internal consistency, and a system can be internally consistent and useless.

The gap between those two things is where a 95 percent becomes a 44 percent a week after launch.

The one decision that matters: what you hold out

Everything else in this article is detail. This is the decision.

A random split scatters every source across both sides. If one speaker, one site, one device, one annotator, or one customer dominates your data, that source is in your training set thousands of times and also in your test set. So the test asks the model to recognise the thing it was built around, using data collected in the same room, on the same equipment, in the same period.

The model's shortcut works perfectly on that test. It scores well because the shortcut is valid on both sides of a line that was drawn without regard to it.

A group split holds out entire units. Every row belonging to a held-out speaker, site, device, or customer goes to the test side and none of it appears in training. Now the test asks whether the model generalises to a new one of those, which is the question production will ask on day one.

Same data, same model, honest number. Frequently a much lower number, which is the point.

Choosing the grouping variable

Pick the unit that will be new in production. That is the whole rule, and it is worth being deliberate rather than defaulting.

If you will deploy to new users, hold out users. New sites or factories, hold out sites. New hardware revisions, hold out devices. New customers in a multi-tenant product, hold out tenants. New document sources, hold out sources. A model that runs forward in time, hold out a later time period, which is the temporal version of the same idea and the only correct split when there is any chance of information flowing backwards.

When more than one of those will be new at once, and usually more than one will be, split on the hardest one and report the others separately. A single number cannot carry several independent generalisation claims.

Nested groups, and the mistake inside them

If a device belongs to a site and a site belongs to a customer, splitting on device leaks at the site level. The held-out device sits in a building whose other devices trained the model, sharing acoustics, lighting, network conditions, and operator habits.

Split at the outermost level you will encounter new instances of. It is more conservative, it produces a worse number, and it is the number that will still be true after launch.

Sizing the holdout honestly

Once you split by group, your effective sample size is the number of groups, not the number of rows.

This surprises people, and it is the most common way a group split gets built correctly and then read incorrectly. Holding out one speaker with 4,000 utterances does not give you 4,000 independent test points. It gives you one, with a lot of measurement inside it. If that speaker happens to be easy, your number is optimistic and you have no way to know.

So count groups. A handful of held-out groups tells you very little, and the variance between them will be larger than the effects you are trying to measure. Report per-group results rather than only the pooled average, because the spread across groups is the actual finding: a model that scores well on eight held-out sites and collapses on the ninth has told you something a single average would have concealed.

If you can only afford a few groups, say so in the same sentence as the number. A figure without its uncertainty invites exactly the confidence that causes this failure.

Check for leakage before you believe anything

A score can be inflated without anything being learned, and there are a few reliable ways it happens.

Duplicate and near-duplicate records. The same document ingested twice, the same event logged by two systems, augmented copies of one original scattered across the split. Deduplicate on content, not on identifier, because identifiers are exactly what differs between two copies of one thing.

Target leakage. A feature that is downstream of the answer. A field populated only after the outcome is known, a timestamp that encodes the label, an identifier assigned by a process that already knew. These produce excellent scores and no capability, and they are found by asking of each feature: at the moment of prediction, would this value exist yet?

Evaluation questions in the training corpus. For a retrieval or generation system, if the test questions appear in what the model was trained or tuned on, it is remembering rather than understanding, and it can pump the score substantially while the underlying capability is unchanged.

Preprocessing fitted before the split. Normalisation constants, vocabularies, imputation values, and encoders computed over the whole dataset carry test-set information into training. Fit them on the training side only, then apply.

The question worth asking of any evaluation, and worth asking out loud: how do we know it learned this rather than memorised it?

Make it look like production, not like your team

Two more properties separate a test set that predicts from one that flatters.

Do not let your own team be the test. If the people who built the system are also the people generating the evaluation data, you are measuring whether it works for the people who built it. Hold real users out, with different voices, different devices, different rooms, different handwriting, whatever the analogous axis is for what you are building. If it only works on its builders, find that out on a quiet Tuesday rather than from customers in launch week.

Match the production mix, including the ugly parts. Field data contains the scanned document, the noisy recording, the partially filled form, the malformed record. Curated datasets tend to have these cleaned out, because whoever assembled them was being helpful. That helpfulness is what removes the cases that will actually fail.

Slice the result, always

One number over a test set repeats the original mistake at a smaller scale, because an average hides the small slice that is failing badly.

Report by group, by input type, by difficulty, by source, by age, by whatever axis your business actually varies along. The slice with the worst score and enough volume to be real is the one that tells you whether you can ship, and it is invisible in the mean.

This is the same operation as slicing a production score to find a broken corner. It is cheaper before launch than after.

Then keep watching the inputs

A test set is a snapshot, and it goes stale the same way everything else does.

The score tells you how the model is doing on the distribution you captured. The inputs tell you whether the world still looks like that distribution. The day your real users stop resembling your training data is the day the score becomes a liar, and that day arrives for every system eventually.

So the last property of a test set that predicts production is that somebody re-runs it on a schedule, against current data, and looks at the slices rather than the average.

FAQ

Why does my model score well in testing and badly in production? Most often because the test set was split at random from the training data, so whatever source dominates your data appears on both sides. The model's shortcut works on that test. A random split is structurally incapable of detecting a group-level shortcut, regardless of how much test data you add.

What is a group split and when should I use it? Holding out entire units, every row belonging to a given speaker, site, device, customer, or time period, so none of that unit appears in training. Use it whenever production will present new instances of that unit, which is almost always. Split on the outermost level you will encounter new instances of.

How big should a held-out test set be? Count groups, not rows. Holding out one speaker with 4,000 utterances gives you one independent test point with a lot of measurement inside it. Report per-group results and the spread between them, not just the pooled average.

How do I check my test set for leakage? Deduplicate on content rather than identifier, check every feature against whether it would exist at prediction time, confirm evaluation questions do not appear in the training corpus, and fit all preprocessing on the training side only. Then ask how you know it learned this rather than memorised it.

Should the team that built the model create the test set? Not alone. If the builders also generate the evaluation data, you are measuring whether it works for the builders. Hold real users out, with different devices, environments, and habits, and include the ugly production cases that curated datasets usually have cleaned out.