Build the Gold Set From the Source, Not From Your Own Output

Mostafa Dhouib·10 min read

Build the Gold Set From the Source, Not From Your Own Output

The short answer. If you build your evaluation set by exporting what the pipeline extracted and correcting what looks wrong, you have not built a reference. You have built a mirror. Every error the pipeline makes confidently and consistently survives into that reference, because nobody reviewing an export has any reason to question a value that looks plausible. The score then tells you the pipeline agrees with itself. A usable reference is derived from the source material independently, by someone working from the document rather than from the output, and it is sampled from the awkward documents rather than the representative ones.

This is the mistake I see most often in extraction work, and it is not made by careless teams. It is made by teams under time pressure who have a perfectly reasonable instinct: we already have the extracted values in a table, so let us start from there and clean it up.

The instinct is reasonable and the result is unusable. Worse than unusable, because it produces a number that looks like evidence.

A reference built from output is a mirror, not a measurement

Think about what happens mechanically when a gold set is built this way.

The pipeline runs over a set of documents and produces fields. Somebody opens the export and goes down it, fixing what looks wrong. The things that look wrong are the obvious failures: an empty cell, a date in the wrong format, a number with a stray character, a value that is clearly nonsense. Those get corrected.

What does not get corrected is everything the pipeline got wrong in a plausible way. The value it pulled from the wrong column of a table that has two similar columns. The figure it took from the prior period rather than the current one. The total it read off a subtotal row. The address it assembled from two overlapping blocks. None of these look wrong in a spreadsheet, because the only way to know they are wrong is to go back to the document and read it.

So the reference inherits exactly the class of error you most need to measure. Systematic, confident, plausible mistakes. The ones that are expensive precisely because nothing downstream flags them.

Then you score the pipeline against that reference and it does well. Of course it does. You built the reference out of its answers.

The failure is structural rather than a question of care. Even a diligent reviewer anchored to a proposed value will accept it far more readily than they would have produced it unprompted. That anchoring is the whole reason the independence condition exists.

Why this one is so hard to see from inside

Three things conspire.

A healthy number does not get interrogated. A low accuracy score triggers investigation. A high one triggers a slide. If the evaluation says extraction is good, the natural next step is to move on, not to audit the evaluation.

The errors it hides are the ones users complain about. This is the tell. When measured accuracy is high and the support queue is full of wrong values, the usual conclusion is that users are hitting unusual documents. Sometimes true. More often the evaluation is blind to the error class by construction.

Correcting an export feels like independent work. The reviewer is reading carefully, making changes, exercising judgement. It feels nothing like rubber-stamping. The dependency is invisible because it operates through what the reviewer never thinks to check.

What a source-derived reference actually requires

One condition, and it is a hard one. The person producing the reference value must not have seen the pipeline's value for that field.

Everything else is practical detail. That single condition is what makes the reference an independent measurement rather than a confirmation.

In practice it means a reference pass where the annotator works from the document itself, with the field definitions in front of them, and records what the document says. Not what the pipeline said the document says. The output is loaded afterwards, and only for comparison.

The second condition, nearly as important and far more often skipped, is that the field definition has to be written down before annotation starts. Most of what people call extraction error is definition error. Which of two dates is the effective date. Whether a total includes tax. Whether a party name is the legal entity or the trading name. Whether an absent field is empty or not applicable. If two careful annotators would disagree on a field, the pipeline cannot be wrong about it in any measurable sense, because there is no single fact to be wrong about.

So the sequence is fixed. Define the fields, annotate from source, then compare. Reversing any two of those steps destroys the measurement.

How to build one without a large annotation budget

The reason teams reach for the export is cost. A full independent annotation pass over a whole corpus is expensive, and the pressure to skip it is real. But a useful reference does not need to be large. It needs to be independent and adversarially sampled, and both of those are cheap properties.

Make it small on purpose. A reference of a few dozen well-chosen documents, annotated properly, is worth far more than a corpus-sized set built from output. You are not trying to estimate population accuracy to a tight confidence interval. You are trying to find out whether the pipeline makes a class of error. For that, a small set of hard cases is the efficient instrument.

Annotate fields, not documents. You do not need every field on every document. Pick the fields that carry decisions downstream and annotate only those. A reference covering the handful of load-bearing fields across a wide spread of documents beats one covering everything across a narrow spread.

Use double annotation only where definitions are shaky. Two independent annotators on every field is the textbook answer and it doubles the cost. Spend it where you expect disagreement, which is usually a small subset of fields, and read the disagreement rate as a diagnostic on your definitions rather than as a quality gate.

Treat the reference as a durable asset. The budget feels large because people price it as a one-off cost against one release. It is not. A properly built reference outlives several model changes, several prompt revisions and at least one vendor swap, and it is the only thing that will tell you whether any of those helped. Amortise it accordingly.

Store the provenance, not just the value. For each reference value, record where in the document it came from. Page, section, table, cell. This turns every disagreement into a short adjudication rather than a re-reading, and it is what keeps the reference maintainable when field definitions inevitably change.

Sample the awkward documents, not a random slice

Random sampling is the wrong strategy here, and that is counterintuitive enough to be worth spelling out.

A random sample is the right tool for estimating a population average. It is the wrong tool for finding out where a system breaks, because the documents that break it are rare by definition. A random slice of a corpus is dominated by the documents the pipeline already handles, so most of the annotation budget goes on confirming that easy cases are easy.

What you want instead is a deliberately awkward sample. Things worth hunting for:

Structural variation. Multi-column layouts, tables spanning pages, rotated pages, nested tables, footnotes carrying the real value, headers that repeat mid-document.

The long tail of sources. If the corpus comes from many issuers, templates or vendors, the common template is well covered and the rare ones are not. Sample by source, not by volume.

Confusable field pairs. Two dates, two totals, two party names, a subtotal sitting next to a total. This is where the plausible-but-wrong errors live, which is exactly the class an output-derived reference cannot see.

Degraded inputs. Scans, photographs, faxes, re-exported files, anything that has been through a lossy step.

Documents users complained about. The support queue is a free adversarial sampler and it is almost always underused. Every complaint is a document somebody has already told you is hard.

A small random stratum anyway. Keep a modest random slice alongside the awkward one, for two reasons. It stops the reference drifting into a set of pathological cases that represent nothing, and it gives you a check that the common path has not regressed.

Then report the two strata separately. A single blended accuracy figure over a deliberately skewed sample is not interpretable. Accuracy on the hard stratum and accuracy on the random stratum are each meaningful, and the gap between them is the most useful of the three.

The honest limit

A source-derived reference tells you whether the pipeline agrees with the document. It does not tell you whether the field definitions are the right ones for the business, and it cannot. If the definition says the effective date is the one on the signature block and the business actually needs the commencement date, a perfect score against a perfect reference is still the wrong answer, and no amount of annotation discipline will surface that. Definition review is a separate conversation with the people who use the output.

It also will not survive carelessness about independence. If annotators have the pipeline's output in an adjacent tab, or if the annotation tool pre-fills suggestions to save time, independence is gone and you are back to a mirror with extra steps. The pre-fill is the most common way this leaks back in, because it is introduced as an efficiency improvement by someone who was not part of the original decision.

And the awkward sampling strategy deliberately gives up the ability to estimate population accuracy. If you need a defensible corpus-wide figure, for a regulator or a contract, that is a different exercise with a different sample design, and a hard-case reference will not substitute for it.

FAQ

Why can't I build my evaluation set by correcting the pipeline's output? Because a reviewer correcting an export only fixes what looks wrong, and the expensive errors look right. A value taken from the wrong column, the prior period or a subtotal row is plausible in a spreadsheet and detectable only against the document. Those errors survive into the reference, so scoring against it measures whether the pipeline agrees with itself.

How large does a gold set need to be? Smaller than most people assume, if it is built correctly. A few dozen properly annotated documents covering the fields that carry decisions will find error classes that a corpus-sized set derived from output cannot see at any size. Size buys precision on a population estimate. Independence and adversarial sampling buy the ability to find faults, which is usually what you need first.

Should I sample documents randomly for my reference set? Not primarily. A random slice is dominated by documents the pipeline already handles, so most of the annotation budget confirms that easy cases are easy. Sample deliberately for structural variation, rare templates, confusable field pairs and degraded scans, then keep a small random stratum alongside it and report the two separately.

What makes a reference value independent? The annotator must not have seen the pipeline's value for that field while producing theirs. That is the whole condition. It is most often broken by annotation tooling that pre-fills suggestions as an efficiency measure, which quietly converts independent annotation back into review of output.

Is disagreement between annotators a quality problem? It is usually a definition problem, and it is informative. If two careful people reading the same document produce different values, the pipeline cannot be meaningfully right or wrong on that field, because there is no single fact to compare against. Treat the disagreement rate as a diagnostic on your field definitions and fix the definitions before scoring anything.

Can a high accuracy score coexist with users reporting wrong values? Routinely, and it is the clearest signal that the evaluation was built from output. The score is blind to the error class by construction, while users hit it constantly because it occurs on the plausible path rather than the obviously broken one. Rebuild the reference from source on a small adversarial sample before investigating anything else.