Score Each Stage or You Are Tuning Blind

Mostafa Dhouib·10 min read

Score Each Stage or You Are Tuning Blind

The short answer. A pipeline with extraction, retrieval, fusion and generation stages has one visible output and four places to be wrong. Several unrelated causes produce the identical symptom: a thin, hedged, incomplete answer. The extraction may have dropped the field. One retrieval branch may be returning nothing. The fusion step may be quietly absorbing an empty list. Or generation may be failing on context that was perfectly good. End-to-end scoring cannot separate these, so every change you make after that score is a guess. The fix is per-stage instrumentation first, attribution second, improvement third.

The symptom is always the same and it is maddeningly uninformative. Somebody asks a question the system should be able to answer. The answer comes back short, cautious, missing half of what it should have said. Nothing errored.

What happens next is usually prompt work, because the prompt is the cheapest thing to change. Then chunking. Then the embedding model. Weeks go by and the symptom moves a little, which feels like progress and is actually noise.

Four causes, one symptom

Walk the pipeline and count the places a thin answer can originate.

Extraction dropped or mangled the field. The source document contained the value and the extracted representation does not, or contains it in a form nothing will match. Downstream, the content simply is not there to retrieve. The generation stage behaves correctly by hedging.

A retrieval branch is dead. If you run a dense branch and a lexical branch, one of them may be returning nothing at all. A bad filter, a stale index, a scope condition that never matches, a query transformation that produces an empty string. The other branch still covers the easy queries, so the overall system looks degraded rather than broken.

Fusion absorbed an empty list. This is the one that gets missed, because fusion code is almost always written to be tolerant. It takes several ranked lists, merges them, and if one is empty it merges the others. That is the correct behaviour for robustness and it is catastrophic for observability, because it converts a hard failure in one branch into a slightly worse result overall.

Generation failed on good context. The right passage was in the context window, in the right position, and the model still did not use it. Context too long, instruction conflict, a passage that contradicts another passage, a format the model will not quote from.

All four of these end with a user reading a disappointing answer. From the output alone, no amount of staring will tell you which one you have. And they lead to completely different work. The first is a parsing problem, the second a configuration problem, the third a design problem and the fourth a prompting or context problem. Only one of the four is fixed by anything a prompt engineer does.

Candidate counts are the cheapest instrument you will ever build

Before scoring, before any gold set, log the sizes.

For every query, record how many candidates each retrieval branch returned, how many survived each filter, how many entered fusion, how many came out of fusion, how many were passed to generation and how many were cited in the answer. One line per query, numbers only.

This is nearly free to implement and it finds an embarrassing proportion of real problems on its own.

A branch returning zero for every query is immediately visible. A branch returning zero for a particular class of query, which is the more common and more interesting case, shows up as soon as you group the counts by query type. A stage receiving fewer candidates than the next stage needs is obvious in a count and invisible in an answer. A reranker configured to return fewer items than the generation prompt expects announces itself.

The specific pattern to watch for is a narrowing that nobody designed. Each stage was built with a sensible local limit, and the composition of those limits leaves the final stage with a handful of candidates. Nobody chose that number. It is the product of four independent decisions, and it is only visible if you log all four.

Two refinements that pay for themselves:

Log the counts at the branch level, not the pipeline level. A total candidate count hides a dead branch completely, because the surviving branches fill the quota.

Log zero explicitly. The most common instrumentation failure I see is code that logs the count only when it is non-zero, usually because the logging sits inside the loop that processes results. An empty branch then produces no log line at all, which reads as absence of a problem.

Scoring stages separately against the reference

Counts tell you where content disappeared. They do not tell you whether the content was right. For that you need a reference, and the reference has to be labelled at stage boundaries rather than only at the output.

Concretely, for each item in your evaluation set, record three things. What the source document says, which is the extraction target. Which passages or records contain the answer, which is the retrieval target. And what a correct answer looks like, which is the generation target.

That lets you score each stage against its own target.

Extraction score. Does the extracted representation contain the value the document contains? This is where a source-derived reference earns its cost, because a reference built from the pipeline's own extraction output agrees with the extraction wherever it is wrong and gives you a perfect score on a broken stage.

Retrieval score. Given the correct extracted content, do the branches return the passages known to contain the answer, and at what rank? Score recall at the candidate count you actually pass forward, not at some conventional cut-off.

Fusion score. Does the merged list retain the known-relevant passages that the individual branches found? A fusion step that drops a passage one branch ranked first is a real and surprisingly common bug.

Generation score. Given the correct passages placed in context, does the answer contain the right content and cite the right source? This is the only stage where you should be evaluating phrasing, and it has to be evaluated on known-good context or you are measuring retrieval again.

The point of the breakdown is not four numbers for a dashboard. It is that each number has an owner and an action. A retrieval recall figure tells you to go and look at indexing and query construction. A generation score on known-good context tells you to go and look at the prompt. An end-to-end figure tells you to go and look at everything, which in practice means guessing.

Attribution before improvement

The discipline that follows from this is simple and consistently unpopular, because it delays the satisfying part.

No improvement work starts until the failure has been attributed to a stage. Every failing case in the evaluation set carries a stage label. If a case cannot be attributed, that is itself a finding, and it usually means the instrumentation is missing at a boundary.

The reason to be strict about it is that unattributed work has a bad expected value. Four plausible causes means a one in four chance of changing the stage that is actually broken, and a change to a working stage does not leave it unchanged. It perturbs it. The symptom moves, somebody reads that as signal, and now you have two problems and a misleading history.

The second reason is that attribution is cumulative in a way that tuning is not. A stage label on every failure, collected over a few weeks, tells you where the system's weight actually sits. Almost always it is concentrated in one stage, and almost always it is not the stage people were working on.

A practical version of the rule: every failing case in the evaluation set must carry a stage label before anyone opens a prompt file. That one sentence has saved more time than any technique in this piece.

The honest limit

Stage-level scoring assumes the stage boundaries are real. In a pipeline where extraction and chunking are entangled, or where query construction happens inside the retrieval call, there is no clean place to measure, and forcing a measurement point in can mean restructuring code purely for observability. Sometimes that is the right trade and sometimes it is not.

The breakdown also will not catch interaction effects. Every stage can score well in isolation and the composed system still underperform, because the output of one stage is subtly mismatched to the assumptions of the next. Chunk boundaries that split a value from its label score fine on extraction and fine on retrieval and ruin generation. A stage-level score does not see that, and only end-to-end evaluation will.

So keep both. The end-to-end number is the one that matters to a user and the stage numbers are the only ones that tell you what to do. Reporting one without the other is how teams end up either tuning blind or optimising four metrics nobody experiences.

FAQ

Why can't I diagnose a thin answer from the output alone? Because at least four unrelated causes produce the same thin answer: extraction dropped the field, a retrieval branch returned nothing, fusion absorbed an empty list, or generation failed on context that was good. The output looks identical in all four cases, and each one requires completely different work, so any change made before attribution is a guess with roughly a one in four hit rate.

What is the cheapest useful instrumentation for a retrieval pipeline? Candidate counts at every boundary, logged per branch rather than per pipeline, with zeros logged explicitly. How many each branch returned, how many survived each filter, how many entered and left fusion, how many reached generation. It takes very little code and it surfaces dead branches and unplanned narrowing that no amount of output inspection will reveal.

How does fusion hide a broken retrieval branch? Fusion code is written to be tolerant, so when one input list is empty it merges the rest and returns a result. That is correct for robustness and destructive for diagnosis, because a hard failure in one branch becomes a mildly worse overall answer. Logging the per-branch input sizes into fusion is what makes the failure visible again.

Can I score extraction against the pipeline's own extracted output? No, and this is the trap that makes stage-level scoring look unnecessary. A reference built from the pipeline's output agrees with the pipeline wherever the pipeline is wrong, so the extraction stage scores near perfectly while being broken. The extraction target has to come from the source document, produced by someone who did not see the pipeline's answer.

Should generation be evaluated on the pipeline's real context or on known-good context? Both, for different purposes. Evaluated on real context, a generation score is mostly a retrieval score wearing a disguise. Evaluated on context you have verified contains the answer, it isolates whether the model uses good material, which is the only configuration in which prompt changes can be judged honestly.

Does per-stage scoring replace end-to-end evaluation? No. Every stage can score well in isolation while the composed system underperforms, because one stage's output can be subtly mismatched to the next stage's assumptions. Chunk boundaries that separate a value from its label are the classic case. Keep the end-to-end number as the thing users experience and the stage numbers as the thing that tells you what to change.