Tagged evaluation
8 pieces
- · 10 min read
Build the Gold Set From the Source, Not From Your Own Output
A reference set assembled from a pipeline's own extracted output scores that pipeline against itself. Wherever the extraction is wrong, the reference is wrong in the same direction, and the evaluation agrees. The reference has to be derived from the source material by someone who did not see the pipeline's answer, and it has to be sampled from the documents that are hard rather than the documents that are typical.
- · 10 min read
Score Each Stage or You Are Tuning Blind
A multi-stage pipeline judged only on its combined output cannot tell you which stage contributed nothing. An extraction error, a dead retrieval branch, a fusion step absorbing an empty list and generation failing on good context all produce the same thin answer. Per-stage instrumentation means candidate counts per branch, per-stage scoring against a source-derived reference, and attribution before any improvement work starts.
- · 10 min read
A Confidence Score That Never Changes Cannot Route Your Review Queue
Calibration in aggregate and the ability to rank individual items are two different properties, and a score can have the first without the second. In a lab reproduction, a calibrated but nearly constant confidence score caught 4 percent of errors at a 5 percent review budget, against 95 percent for an informative score on the same data. Threshold tuning cannot repair a score that does not rank, because there is no threshold at which a flat score separates anything.
- · 7 min read
You Can Be 90 Percent in the Green Zone and Measuring Nothing
A model that predicts close to the population mean can place most of its points in a clinical error grid's acceptable zones and post a respectable error percentage, while explaining less variance than the mean itself. Check the variance explained against a mean baseline, under a subject-disjoint split, before believing any accuracy figure from a physiological model.
- · 8 min read
Price the Baseline in the Same Pass, or Your Ranking Is Not Answering the Question
A product that offers a clever alternative to an obvious option has to show it beats that option, on the same inputs, at the same moment. Ranking your own candidates against each other answers which of yours is best, which is not what the user asked. The baseline has to be computed in the same pass, or the comparison is against a number from a different time with different availability.
- · 7 min read
Model Confidence or Independent Verification
A confidence threshold is the cheapest thing that looks like a safety control, and it fails in the one place you need it: models are most confident on inputs they have never seen. The decision is not what threshold to set, it is whether your task has a cheap grounded check, because that determines whether verification is available at all.
- · 7 min read
Questions to Ask Before Hiring Anyone to Fix Your RAG System
Seven questions, in the order that sorts candidates fastest. The first three separate people who have run a retrieval system in production from people who have read about one, and the last one beats any pitch, because someone who has done this work can tell you exactly where their approach falls down.
- · 8 min read
The Verifier That Verifies Nothing
Naming a pipeline stage 'verifier' does not make it one. If the stage is another model call asked whether the output looks right, it shares the generator's blind spots, produces an opinion rather than a check, and has almost certainly never rejected anything. There is a ladder of verifier strength with five rungs, and most production systems are on the bottom one while believing they are near the top.