Tagged extraction
3 pieces
- · 10 min read
Build the Gold Set From the Source, Not From Your Own Output
A reference set assembled from a pipeline's own extracted output scores that pipeline against itself. Wherever the extraction is wrong, the reference is wrong in the same direction, and the evaluation agrees. The reference has to be derived from the source material by someone who did not see the pipeline's answer, and it has to be sampled from the documents that are hard rather than the documents that are typical.
- · 8 min read
Schema-Valid and Ten Thousand Times Wrong: Units Are Semantics, Not Syntax
A strict JSON schema on an extraction pipeline validates type, range and required fields. It cannot see that the number 12 in a table headed in ten-thousands means 120,000. The row passes every check, lands in the database as 12, and is wrong by four orders of magnitude with no validation error anywhere. The fix is an arithmetic cross-check against a total the document already states.
- · 8 min read
The Number Someone Signs: Rebuild It by Rule, Do Not Read It Off the Document
When a person certifies a figure your system produced, extraction accuracy is the wrong metric. The document your pipeline reads can be perfectly extracted and still contain a total that must not be claimed, because a rule outside the document excludes part of it. The deliverable is a figure rebuilt by classification from the line items, with each charge categorised, and scored against what was actually filed rather than against what was extracted.