Hybrid search and model-based extraction over documents full of tables, written in a language the standard text-search stack has no support for. The visible request was better search quality. The failure that mattered sat at the joint between extraction and the index, where a value can be perfectly well formed and still wrong, and every stage after it inherits the error without complaint.
Hybrid retrieval runs two searches and blends the results, so one of them can stop working without anyone noticing.
Document search systems now commonly combine two kinds of search. Vector search turns text into numerical embeddings and finds passages whose meaning is close to the query, even when the words differ. Lexical search matches the actual words, which matters for names, codes, product identifiers and figures, the things embeddings tend to blur. A hybrid system runs both and merges their ranked lists. A common way to merge them is rank fusion, where each document earns credit from its position in each list and the credits are added together.
Lexical search depends on a text-search configuration for the language: how to split text into words, how to reduce words to a common form, which words to ignore. For widely used languages these come built in. For a language without one, text is processed by a generic fallback that does not split or normalise it in a useful way, and queries can match nothing at all. Rank fusion then receives one full list and one empty list. An empty list adds no credit to anything, so the fused ranking is the vector ranking, unchanged. The system still reports itself as hybrid, and no step raises an error.
Tables add a second trap at extraction time. A model reads a table and emits structured rows, and a schema checks that each field has the right type: an amount is a number, a date is a date. Schema checks are about form. They cannot tell that a column header said the figures are in ten-thousands, so a bare number in a cell passes validation while being wrong by that whole factor. The defence comes from the documents themselves. Financial and official tables usually state totals, and a total is an independent check on every row that feeds it.
Two of the obvious moves make things worse. A gold set built from extracted rows grades the pipeline against its own output, so it agrees with itself wherever it is wrong. It has to be built from the source tables. A score taken only on the fused result cannot say which branch contributed nothing, so each stage has to be scored on its own. Both silent failures here are of that kind: a lexical branch that returns no rows at all, and an extracted amount that passes every schema check while being wrong by four orders of magnitude.
Multi-stage pipelines fail in the stage nobody scored separately.
Retrieval pipelines are usually evaluated where the user meets them, on the final answer. That is a sensible place to judge quality and a poor place to find causes. A weak answer can come from extraction that got a value wrong, from a branch that contributed nothing, or from ranking that put the right passage too low. From the final output these look the same: a thin answer.
The second pattern is a reference built from the system's own work. Labelling extracted rows is far quicker than going back to the source tables, so gold sets are often built that way. Every extraction error is then copied into the answer key, and the evaluation reports agreement exactly where the pipeline is wrong.
The reflex, and the fix.
Tune the fused ranking
It is where quality is judged, and the fusion weights are the obvious knob.
If one branch contributes nothing, no weighting of it changes the result. The tuning is spent on a parameter that is multiplying zero.
Score each stage against a gold set from source
Building a gold set from the source tables and planting faults on purpose, before any tuning.
Each stage attributable on its own, so a dead branch or a unit error becomes a measurement instead of a theory.
Attribute before improving, because three different causes produce the same thin answer.
A gold set built from the source tables
The reference answers come from the original tables, never from extracted rows. A gold set built from the pipeline's own output scores the pipeline against itself and agrees with it wherever it is wrong.
The lexical branch verified separately
The standard recipe pairs a vector index with a full-text index. For a language the text-search stack has no configuration for, the lexical side returns nothing, and rank fusion absorbs that silently. The result is the dense ordering, presented as hybrid, with no error raised anywhere. This was shown in a lab reproduction of the commonly published recipe.
Units extracted, never assumed
A table headed as being in ten-thousands reports bare numbers in its cells. A strict schema accepts the bare number, and the value is then wrong by a factor of ten thousand with no validation error anywhere. The unit is part of the value and is extracted as such.
Arithmetic cross-checks against stated totals
Documents of record are internally redundant, because people check them too. Summing the line items and comparing them with the stated total catches unit errors, missed rows and duplicated rows in one comparison.
Planted faults
Known faults are injected into the gold set, so the harness is shown to detect the classes of error it claims to detect. A harness that has never caught a planted fault has not been tested.
What each choice cost, and what it bought.
ABuilding the gold set from the source tables
Going back to the original tables is slower than labelling extracted rows, and it surfaces disagreements that then have to be resolved by reading the source carefully. It is the only reference that can catch an extraction error, because it was made without the extractor. A gold set built from output would have scored the unit error as correct.
BScoring each stage on its own
Extraction was scored against the source, and each retrieval branch was measured for what it actually contributed before anything was fused. That adds several evaluations where one would usually be run. It is what turned an empty lexical branch from a suspicion into a measurement, and it stopped effort going into fusion weights that could not have helped.
CTreating the unit as part of the value
Units and scale were extracted explicitly from headers and notes, and every extracted table was checked arithmetically against its stated total where one existed. That adds work to extraction and leaves documents with no stated total flagged for attention. It catches the class of error a schema is structurally blind to, along with missed and duplicated rows, in a single comparison.
DPlanting faults before trusting the harness
Known errors, of the classes the harness claims to detect, were placed into the gold set deliberately. If the harness failed to flag them it would have been fixed before it was used to judge anything. That is extra work spent on the measuring instrument. Without it, a clean report could mean a clean pipeline or a blind harness, and there is no way to tell which.
- Gold set from source tablesmade without the extractor
- Plant known faultsprove the harness can see them
- Score extraction aloneagainst source, units included
- Measure each branch's contributiondoes lexical return anything at all
- Cross-check against stated totalsunits, missing rows, duplicates
- Only then tune fusionon inputs known to be live
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
Reproduction results are lab measurements with their method and versions stated. They are not client outcomes.
A hybrid system running dense-only without anyone knowing, tuned on its fusion weights, serving values that are schema-valid and four orders of magnitude out.
A gold set from source, each stage scored on its own, the lexical branch confirmed to contribute, units extracted explicitly and cross-checked against stated totals, and planted faults proving the harness sees what it claims to.
What it owns, and what it hands to a person.
The reproductions behind this were lab measurements, run locally on a stated database and extension version against the commonly published recipe. They are not client outcomes. The behaviour follows from how text-search configurations and rank fusion each work, with no defect in either, so it should hold broadly. It was measured on one configuration.
Any multi-stage pipeline whose stages are judged by one combined output. Build the reference from the source, measure what each stage contributes, and plant faults so the harness proves it can see them.
A gold set built from a pipeline's own output agrees with that pipeline wherever it is wrong. Build the reference from the source.
In hybrid search, verify that each branch returns candidates on its own. Rank fusion absorbs an empty branch silently, and the system keeps calling itself hybrid.
Lexical search depends on a text-search configuration for the language. Without one, it can return nothing at all and raise no error.
A schema checks the form of a value, never its meaning. Extract units and scale explicitly, and use stated totals to check the arithmetic.
Plant known faults in your evaluation before trusting it. A harness that has never caught anything has not been tested.