Tool

Can your test set detect a shortcut?

A random split measures how much your test data resembles your training data. If one source dominates, that split is structurally incapable of catching the failure that will actually kill you in production.

Samples, utterances, documents, images, whatever your unit is.
Speakers, sites, devices, tenants, collectors, time periods.
%
The share of the data from the single biggest source. Slice and look before guessing.
%
This calculator runs entirely in your browser. Nothing you type is sent anywhere unless you ask for the result by email at the bottom of the page.
What a random split does with that
Dominant group in training
~22,400
rows
Dominant group in test
~5,600
the same source, on both sides
Every other group averages
2.7%
of the data, each
One source holds 70.0% of your data and a random split puts it on both sides of the line. A model that learns that source's fingerprint instead of the task scores well on this test, and the test cannot detect it: the shortcut is valid on both sides. Adding more test data drawn the same way changes nothing.
What a group split gives you instead
Effective sample size
2 groups
not 8,000 rows
Held out
2 of 12
whole groups, none of them in training
Holding out 2 groups gives you 2 independent test points, not 8,000. That is not enough to measure anything: if the held-out group happens to be easy, your number is optimistic and you have no way to know. Collect more groups, or report the result with that limitation stated in the same sentence.
Send me this result, with how I would split it

Your numbers go with it. You get a straight technical reply on the grouping variable I would hold out for a dataset shaped like this, and what to check for leakage.

Your inputs are included so the reply can be specific.

The arithmetic, so you can check it

What a random split actually does. It scatters every source across both sides in proportion to its share. So a source holding seventy percent of your data appears roughly seventy percent of the time in training and seventy percent of the time in test, in the same room, on the same equipment, in the same period.

A model that learns that source\u2019s fingerprint instead of the task scores well on that test. Not because the test is weak or too small, but because the shortcut is valid on both sides of a line drawn without regard to it. That is what makes the failure structural rather than unlucky: no amount of additional test data drawn the same way changes it.

What a group split gives you. Hold out whole units and the test asks whether the model generalises to a new one. Same data, same model, an honest number, and frequently a much lower one, which is the point.

The part that gets read wrong. Once you split by group, your effective sample size is the number of groups, not the number of rows. This is the most common way a correct split is then reported incorrectly. Held-out groups in single digits tell you very little, and the variance between them will exceed the effect you are measuring.

Which unit to hold out

Whichever will be new in production. New users, hold out users. New sites, hold out sites. New hardware revisions, devices. New tenants, tenants. A model that runs forward in time, a later period. When several will be new at once, split on the hardest and report the others separately, because one number cannot carry several independent generalisation claims.

The full treatment, including the leakage checks, is in how to build a test set that predicts production.

Questions

Why does my model score 95 percent in testing and fail on real users?

Most often because the test set was split at random, so the source that dominates your training data also appears in your test set. A model that learned that source's fingerprint rather than the task scores well on that test, and the test cannot detect it because the shortcut is valid on both sides of the line.

What is a group split?

Holding out entire units, every row belonging to a given speaker, site, device, tenant, or time period, so none of that unit appears in training. The test then measures generalisation to a new one of those, which is the question production asks on day one.

How large does a held-out test set need to be?

Count groups, not rows. Holding out one speaker with 4,000 utterances gives you one independent test point with a lot of measurement inside it, not 4,000. Report per-group results and the spread between them rather than the pooled average.

Will adding more test data fix a shortcut problem?

No. If the split strategy puts the dominant source on both sides, more data drawn the same way reproduces the same blind spot at greater expense. The split strategy is the problem, not the sample size.