A group is whatever will be new in production: a speaker, a site, a device, a tenant, a patient. Pick the outermost level you will encounter new instances of, because splitting below it leaks.
| Computed on | Effective n | 95% interval | Half-width | What it means |
|---|---|---|---|---|
| Quoted on rows | 10,000 | 91.5% to 92.5% | +/- 0.5% | a fiction |
| Honest, on groups | 3 | 43.8% to 100.0% | +/- 28.1% | what a new group gives you |
| Per slice | 1 | 20.7% to 100.0% | +/- 39.7% | 5 slices of the held-out set |
Interval half-width, by held-out groups
The 95 percent Wilson interval on an observed accuracy, computed on the number of held-out groups. Real datasets usually live in the first two or three rows, and the widths there are not survivable by any precise claim. Note that higher accuracy narrows the interval, which is why a system near 99 percent can make do with fewer groups than one near 80.
| Held-out groups | 80% observed | 90% observed | 95% observed | 99% observed |
|---|---|---|---|---|
| 3 | +/- 36.5 pts | +/- 28.1 pts | +/- 28.1 pts | +/- 28.1 pts |
| 5 | +/- 29.4 pts | +/- 21.7 pts | +/- 21.7 pts | +/- 21.7 pts |
| 10 | +/- 22.7 pts | +/- 19.3 pts | +/- 13.9 pts | +/- 13.9 pts |
| 20 | +/- 16.8 pts | +/- 13.7 pts | +/- 11.4 pts | +/- 8.1 pts |
| 50 | +/- 10.9 pts | +/- 8.5 pts | +/- 6.2 pts | +/- 3.6 pts |
| 100 | +/- 7.8 pts | +/- 6.0 pts | +/- 4.5 pts | +/- 2.6 pts |
| 250 | +/- 4.9 pts | +/- 3.7 pts | +/- 2.7 pts | +/- 1.3 pts |
| 500 | +/- 3.5 pts | +/- 2.6 pts | +/- 1.9 pts | +/- 0.9 pts |
Red is wider than plus or minus 10 points, amber wider than 3. A claim of 92 percent from 10 held-out groups means somewhere between roughly 65 and 99 percent, which supports very little.
A worked example
A speech system with 40,000 utterances from 12 speakers, split 25 percent for test, reporting 92 percent accuracy across five slices. The evaluation looks substantial: ten thousand held-out utterances is a lot of test data by any ordinary standard.
Computed on rows, the 95 percent interval is 91.5 to 92.5 percent, or plus or minus half a point. That is the number that gets into the deck. Computed on the three held-out speakers, which is the effective sample size once you split by group, it is 43.8 to 100 percent. The honest interval is 53 times wider, and it is the one that describes what happens with a speaker nobody has met.
The per-slice numbers are worse again. Three held-out speakers across five slices is 0.6 groups per slice, so the slice figures are not measurements at all. Ranking them, or acting on the worst one, is reading structure out of noise.
Reaching plus or minus 2.5 points at this accuracy would take 455 held-out groups, implying roughly 1,800 speakers in total. That is not a realistic ask, and stating it is not defeatism: the useful response is to report the wide interval, stop making precise claims the design cannot support, and put the collection effort into more speakers rather than more utterances per speaker.
The arithmetic, so you can check it
Both intervals are Wilson score intervals at 95 percent, which is preferred over the normal approximation because it behaves at small n and near the ends of the range, and small n near the top is exactly where group splits live. The only difference between the two rows is what goes in for n: held-out rows for the fiction and held-out groups for the honest one.
Per slice, the held-out groups divide by the number of slices, because a slice is a subset of the same held-out set rather than a new sample. The groups needed for a target width is found by searching upward until the Wilson width closes to the target at your observed accuracy.
Treating each group as one observation is conservative, and the truth sits between the two intervals shown. Rows within a group carry some independent information, so a properly specified hierarchical model would give something narrower than the group interval and far wider than the row one. If the decision is close enough that the difference matters, fit the hierarchical model. If it is not, use the group interval, because being approximately right about your uncertainty beats being precisely wrong.
Why a random split cannot detect a shortcut is in why AI works in testing and fails in production, and the split design itself in a test set that predicts production. For what a random split does to a dominant group, use the split strategy checker.
Questions
The number of held-out groups, not the number of held-out rows. Rows within a group are not independent observations of the thing you are measuring: they share a speaker, a device, a site, a set of habits. Four thousand utterances from three speakers gives you three independent test points with a great deal of measurement inside each.
Because it assumes independence that a group split explicitly denies. Computing it on rows produces an interval so narrow it implies a precision the design cannot deliver, and every decision anchored to that number inherits the error. The honest interval is often an order of magnitude wider, which is usually wide enough to change what the headline number supports.
Far more than most datasets have. At an observed accuracy near 92 percent, reaching a 95 percent interval of plus or minus 2.5 points takes several hundred held-out groups. If that is not available, the correct response is to report the wide interval rather than the narrow one, because a number without its uncertainty invites exactly the confidence that causes the failure.
The held-out groups divide again. Five slices of twelve held-out groups is a little over two groups each, and an interval on two groups is wide enough that the slices cannot be ranked against each other at all. Reporting slices is right; reporting them without their intervals turns noise into a finding.
Then a random split puts it on both sides, and the split becomes structurally incapable of detecting a shortcut keyed on that group. Not too small and not badly chosen: incapable. More test data drawn the same way changes nothing, and the only fix is to split by group and accept the lower number that results.