Price the Baseline in the Same Pass, or Your Ranking Is Not Answering the Question

Mostafa Dhouib·8 min read

Price the Baseline in the Same Pass, or Your Ranking Is Not Answering the Question

The short answer. If your product exists because it finds something better than the obvious option, then the obvious option is part of the output, not context. Ranking your candidates against each other tells the user which of your answers is best. It does not tell them whether any of them is worth having, and that is the only question they actually have. The baseline has to be computed in the same pass, from the same inputs, at the same moment, because a comparison against a cached or separately fetched baseline is a comparison against a different point in time with different availability. On a self-transfer flight search, ranking after a viability gate and against a live nonstop baseline took the share of searches returning the cheapest genuinely valid result from 81 to 95.8 percent.

There is a category of product whose whole proposition is relative. A self-transfer flight search beats a nonstop. A refinancing tool beats your current rate. A consolidation service beats your existing arrangement. A routing optimiser beats the default route.

In every one, the user's question is not "which of these is best". It is "is any of this better than what I would have done anyway".

Internal ranking answers the wrong question

The default implementation generates candidates, scores them, sorts, and returns the top ones. That ordering is internally coherent and it is silent on the only comparison the user cares about.

Three things go wrong when the baseline is missing.

The product can confidently return something worse than doing nothing. The top result is the best of your candidates, and the best of a bad set is still bad. Nothing in the pipeline notices, because nothing computed what "bad" means in this context.

Your own metrics mislead you. You measure coverage, result counts, latency, click-through. None of those distinguishes a search that found a genuinely cheaper route from one that found the least-bad of several worse ones. The product looks healthy while delivering negative value.

Users find out on their own, once. Somebody checks the obvious option separately, sees it is cheaper, and stops trusting the product. That is not a conversion problem, it is a credibility problem, and it does not recover.

The baseline has to be in the same pass

The baseline is a priced option, so it has to be priced. The question is when.

Computed in the same pass, with the same parameters, against the same provider state, it is directly comparable. The comparison means what it appears to mean.

Computed separately, from a cache or a periodic job, it is not. Availability moves, prices move, and the gap between your candidate and a baseline from an hour ago is partly the gap and partly the hour. Worse, the error is not random: baselines tend to be cached because they are expensive to refresh, and expensive to refresh usually means volatile, so the stale baseline is systematically wrong in the cases that matter most.

There is a cost argument against this, and it is real. The baseline is another paid call per search, on a path where paid calls are the thing you are rationing. It is worth the slot, because without it the output is not an answer. Budget for it as a fixed overhead rather than as a candidate competing for the same budget as the alternatives.

Gate before you rank, and the ordering compounds

Two things have to happen before the comparison is meaningful, and their order matters.

Candidates that cannot actually be executed are removed first. Then the survivors are ranked against the baseline.

If you rank first and filter afterwards, the top of the list is enriched for options that are attractive precisely because they do not work, since whatever makes them infeasible is usually what makes them score well. Filtering after ranking is mitigation. Filtering before it is design.

On the flight product this ordering did two things at once. The viability gate caught 76 percent of the combinations that could not be made, and false-viable connections fell from 12.4 to 1.9 percent. And because the ranking was no longer being won by options that could not be executed, the share of searches returning the cheapest genuinely valid result went up rather than down, from 81 to 95.8 percent. That number is the one to keep: gating made the product better at its actual job rather than more conservative.

Show the comparison, do not just use it

Once the baseline is computed, displaying it is nearly free and it changes what the product is.

Showing the saving is the value proposition, stated in the user's own numbers. Not a marketing claim, the specific delta for this request.

Returning nothing is a legitimate answer. If nothing beats the baseline, the honest output is that the obvious option is the best one here. A product that says so is more trustworthy than one that always finds something, and the user who gets that answer comes back.

It gives you the metric you actually need. The share of requests where you beat the baseline, and by how much, is the only health measure that tracks delivered value. Result counts and latency do not.

Where this shape appears

Anywhere the product is an alternative rather than a primary.

Price comparison of any kind, where the baseline is the incumbent supplier. Lending and refinancing, where it is the current rate. Energy and insurance switching. Logistics optimisation, where it is the default route or carrier. Procurement tools, where it is the existing contract. And internally, any recommendation engine whose alternative is the status quo, which is almost all of them.

In each case the test is the same. If your system returned its top result and the user then checked the obvious option themselves, would they be glad they used you? If your pipeline cannot answer that, it is not computing the thing it is for.

The honest limit

A baseline priced in the same pass is comparable on price and availability at that instant. It is not a complete comparison, because the alternative usually differs on dimensions that are not in the number: protection, flexibility, risk, effort, recourse if something goes wrong.

A self-transfer itinerary that is cheaper than the nonstop is cheaper and carries no protection if the first leg is late. That is a real difference and the price comparison does not contain it. Being explicit about what the comparison covers is part of making it honest, and a product that shows a saving while silently omitting the dimension where it is worse has solved the ranking problem and created a different one.

FAQ

Why does ranking my own candidates against each other not answer the user's question? Because the user is asking whether any of your results beats what they would have done anyway, not which of yours is best. An internally coherent ranking can confidently return the least-bad of several options that are all worse than the obvious choice, and nothing in the pipeline notices.

Why must the baseline be computed in the same pass? Because availability and price move. A baseline fetched separately or from a cache is a comparison against a different moment, and the error is not random: baselines get cached because they are expensive to refresh, and expensive to refresh usually means volatile, so the stale one is systematically wrong in exactly the cases that matter.

Is the extra call for the baseline worth the cost? Yes, and it should be budgeted as a fixed overhead rather than competing with candidates for the same allowance. Without it the output is not an answer to the question asked, so the rest of the paid calls are being spent on something that cannot be evaluated.

Should infeasible options be filtered before or after ranking? Before. Options that are attractive because they do not work score well, so ranking first puts them at the top and filtering afterwards is mitigation. On one product, gating first caught 76 percent of the combinations that could not be made, and still raised the share of searches returning the cheapest valid result, from 81 to 95.8 percent.

What should the product do when nothing beats the baseline? Say so. Returning the obvious option as the best answer is honest, it is more trustworthy than always finding something, and it gives you the metric that matters: the share of requests where you genuinely beat the baseline and by how much. Result counts and latency do not track delivered value.

What does a price comparison against a baseline leave out? Everything not in the number. Protection, flexibility, recourse and effort usually differ, and a cheaper option that carries no protection is cheaper and worse on a dimension the comparison does not contain. Stating what the comparison covers is part of making it honest.