The arithmetic, so you can check it
What a random split actually does. It scatters every source across both sides in proportion to its share. So a source holding seventy percent of your data appears roughly seventy percent of the time in training and seventy percent of the time in test, in the same room, on the same equipment, in the same period.
A model that learns that source\u2019s fingerprint instead of the task scores well on that test. Not because the test is weak or too small, but because the shortcut is valid on both sides of a line drawn without regard to it. That is what makes the failure structural rather than unlucky: no amount of additional test data drawn the same way changes it.
What a group split gives you. Hold out whole units and the test asks whether the model generalises to a new one. Same data, same model, an honest number, and frequently a much lower one, which is the point.
The part that gets read wrong. Once you split by group, your effective sample size is the number of groups, not the number of rows. This is the most common way a correct split is then reported incorrectly. Held-out groups in single digits tell you very little, and the variance between them will exceed the effect you are measuring.
Which unit to hold out
Whichever will be new in production. New users, hold out users. New sites, hold out sites. New hardware revisions, devices. New tenants, tenants. A model that runs forward in time, a later period. When several will be new at once, split on the hardest and report the others separately, because one number cannot carry several independent generalisation claims.
The full treatment, including the leakage checks, is in how to build a test set that predicts production.
Questions
Most often because the test set was split at random, so the source that dominates your training data also appears in your test set. A model that learned that source's fingerprint rather than the task scores well on that test, and the test cannot detect it because the shortcut is valid on both sides of the line.
Holding out entire units, every row belonging to a given speaker, site, device, tenant, or time period, so none of that unit appears in training. The test then measures generalisation to a new one of those, which is the question production asks on day one.
Count groups, not rows. Holding out one speaker with 4,000 utterances gives you one independent test point with a lot of measurement inside it, not 4,000. Report per-group results and the spread between them rather than the pooled average.
No. If the split strategy puts the dominant source on both sides, more data drawn the same way reproduces the same blind spot at greater expense. The split strategy is the problem, not the sample size.