The Fan-Out Budget: Expand, Filter With Something Cheap, Spend the Paid Call Last

Mostafa Dhouib·8 min read

The Fan-Out Budget: Expand, Filter With Something Cheap, Spend the Paid Call Last

The short answer. Some products turn one user action into many calls against a provider that charges per call. A flight search across intermediate airports, a shipping quote across carriers and service levels, an availability check across suppliers. The cost of serving that request scales with the number of candidates, not with the number of users, which breaks the usual intuition that cost follows traffic. The structure that fixes it is an ordering rather than an optimisation: generate candidates from data you already hold, eliminate most of them with something that costs nothing, and spend the paid call only on what survives. On a self-transfer flight search that ordering produced a 4.5 second average and an 8.7 second ninety-fifth percentile while holding a per-search budget, with a 35 percent cache hit rate on the filter stage.

Most products have a comfortable cost model. A request arrives, some work happens, the cost of that work is roughly constant, and the bill scales with traffic. You can reason about it with a single number: cost per request.

Then there are products where one request fans out. The user asks one question and answering it well means asking a provider many questions, where each one costs money and takes time.

The moment you are in that shape, cost per request stops being a property of your system and becomes a property of a decision you are making inside it, many times, per request.

Why the obvious version does not work

Take the search literally. The user wants the cheapest route from A to B. Cheap routes might go through any of several dozen intermediate points. So price all of them, compare, return the best.

This is correct and it is unservable.

The cost is multiplicative. Forty candidates means forty paid calls for one search. At realistic provider pricing, a single search can cost more than the transaction it might produce, which means the product loses money in proportion to how useful it is.

The latency is the slowest call, at best. Run them concurrently and you wait for the worst one. Run them in sequence and you wait for the sum. Neither is inside consumer patience at forty candidates.

The rate limits bite. Multiply forty calls by concurrent users and you are above any sensible provider quota within a few simultaneous searches, at which point your failure mode is not slowness but refusal.

And the naive mitigations all cost you the product. Fewer candidates means missing the cheap route, which is the entire reason anyone uses the thing. Caching results means serving stale prices, which in a booking flow is worse than being slow.

The ordering

The structure that works has three stages, and the value is in the order rather than in any one of them.

Generate candidates from something you own. A route graph, corridor rules, a static table of which intermediate points are plausible for this origin and destination. This stage costs nothing because it touches no provider. It should be generous: it is deciding what is eligible, not what is good.

Filter against something cheap. A cached price feed, a historical average, a previous day's data. Something indicative, which you are allowed to be wrong about, whose only job is to rank candidates so the obviously hopeless ones are dropped. This is where the elimination happens and it must not cost a paid call.

Spend the paid call on survivors. Whatever is left gets a live, authoritative query. This is the only stage that costs money, and by the time you reach it the candidate set is a fraction of what you generated.

On the flight search, the filter had a 35 percent hit rate, which is the measurement that matters for this stage: a third of candidates were resolved without touching a provider.

The number to hold is per request

This is the part that changes how you build rather than how you think.

Set a budget for the whole user request. Not per call, not per provider, per request. Then the orchestrator's job is to stay inside it, which means it needs to know how many paid calls it is allowed to make and behave sensibly when it would exceed that.

Behaving sensibly means returning the best answer it found within the budget, with fewer candidates priced, rather than either blowing the budget or failing. A search that priced twelve candidates instead of nineteen is a slightly worse search. A search that cost four times its budget is a business problem, and a search that returned an error is a lost user.

That decision belongs in one place, which is why this is an orchestration concern rather than something each call site can handle.

Bound the concurrency too

Running the surviving calls concurrently is right, and unbounded concurrency replaces one problem with another.

Without a ceiling, a wide search opens as many connections as it has candidates. Under several simultaneous users that exhausts a connection pool, trips a provider's rate limit, or both. The failure is then a refusal rather than a delay, and refusals are much worse: they are visible to the user, they can trigger provider-side penalties, and they arrive in bursts precisely when traffic is highest.

With a ceiling, the same load produces a slower search. The work queues, the response takes longer, and everything still completes. Converting a failure mode into a latency mode is almost always the right trade in a user-facing path.

Where the generalisation lands

The shape is: one request, many candidates, a paid oracle, and a budget.

Shipping and logistics. Carriers times service levels times packaging options, each a rated quote.

Insurance and lending. Multiple underwriters or lenders, each a priced decision.

Travel beyond flights. Properties times date ranges times rate plans.

Enrichment and verification. Records to check against a per-lookup provider.

Model-backed features. The paid oracle is a model call, candidates are prompts or documents, and the economics are identical. Generating candidates cheaply, filtering with embeddings or heuristics, and calling the model only on survivors is the same pattern wearing different clothes.

In every one, the same question applies: what is the cheapest thing that can eliminate a candidate, and is it running before the expensive thing?

The honest limit

The cheap filter is wrong sometimes. That is what makes it cheap.

A candidate eliminated by a stale cached price might have been the best answer. You will not know, because you did not pay to find out. That is a real cost of this design and the only defence is to measure it: periodically price a sample of eliminated candidates and check how often the filter was wrong. If it is dropping winners, loosen it.

The cached value also must never be presented as a quote. It is a gate. The live call is the only authoritative number, and any design where an indicative value can reach a user as a commitment has a different and worse problem than cost.

FAQ

Why does my cost per request explode when I add more providers or options? Because the cost scales with candidates rather than with traffic. One user request becomes many paid calls, so the bill grows with how thorough the search is, not with how many users you have. Cost per request stops being a system property and becomes the result of a decision made many times inside each request.

How do I search widely without paying for every candidate? Order the stages. Generate candidates from data you already own, eliminate most of them with something free such as a cached or historical value, and spend the paid call only on survivors. The elimination stage is what makes the width affordable, and it must not itself cost a call.

What should a fan-out budget be measured against? The whole user request, not the individual call or provider. The orchestrator then returns the best answer found within that budget, pricing fewer candidates when it has to, rather than exceeding the budget or failing. That decision has to live in one place rather than at each call site.

Should I run the provider calls concurrently? Yes, with a ceiling. Unbounded concurrency exhausts connection pools and trips provider rate limits under simultaneous users, which turns a latency problem into a refusal. A bounded fan-out degrades into a slower search instead, and converting a failure mode into a latency mode is almost always right in a user-facing path.

What if the cheap filter eliminates the best candidate? It will, sometimes, and that is the cost of the design. Measure it by periodically pricing a sample of eliminated candidates and checking how often the filter was wrong. If it is dropping winners at a meaningful rate, loosen it. What you must not do is let the cached value reach a user as a quote.

Does this apply to model-backed features? Directly. The paid oracle is the model call and the candidates are prompts, documents or records. Generating candidates cheaply, filtering with embeddings or heuristics, and calling the model only on survivors is the same ordering, and the per-request budget is the same control.