AI InfrastructureField engagement

A comparison engine sounds simple. The layer under it, isolation and run cost, is where it is won or lost.

Architecture and delivery lead.
zero
cross-tenant leaks under adversarial testing
a few dollars
per tenant per month
150 s
the retry the API had to survive
model ~5%
systems engineering ~95%
intelligence on top
drivers, protocols, control plane, distributed telemetry, scale
The model is the small part. The platform underneath, drivers to telemetry to scale, is the system.
What was at stake

Private per-customer similarity search over an API, where each tenant has private data, the criteria are user-defined, and the system has to stay fast and cheap over ordinary HTTP and JSON. Cross-tenant leakage is a security failure rather than a bug, and per-tenant isolation done naively multiplies cost until it hits a hard ceiling while the business is still small.

The constraint

The stated ask is a comparison engine. The real problem is the layer under it: keeping each customer's data walled off, keeping the run cost below what the feature earns, and surviving the front-end integration limits. One of those limits was concrete and non-negotiable: a front end that auto-retries any API call past 150 seconds, which forces idempotency on every endpoint whether or not anyone planned for it.

The fork

The reflex, and the fix.

Road not taken

A separate database per tenant

Pull

It is the intuitive way to guarantee isolation, and it is easy to explain to a security reviewer.

Why not

It multiplies cost per tenant and hits a ceiling early, so the architecture that felt safest becomes the reason the product cannot grow.

Road taken

One shared store with a filter enforced below the app layer

Accepted

Enforcing the per-tenant filter on every read beneath the application, plus a rerank stage so user-defined criteria never force a re-embed.

Bought

Hard isolation, run cost held to a few dollars per tenant per month, and criteria that can change without recomputing anything.

Decision

Isolate below the application layer and put the variable criteria in rerank, because both costs compound with tenants.

How it was built
01Embed once
02Retrieve
03Rerank
04Filter
05Serve
01

Embed the catalog once

Precompute rather than embedding per query. This is cents to dollars, not the cost most people picture when they imagine the bill for a similarity feature.

02

Retrieve, then rerank

User-defined criteria live in the rerank stage, so a customer changing what they care about never forces a re-embed of the catalog.

03

Isolation enforced below the app layer

One shared store with a per-tenant filter applied on every read, beneath the application, so an application bug cannot produce a cross-tenant read.

04

Feedback without a model per tenant

A global base ranker plus per-tenant feature weighting, with graceful fallback until a tenant accumulates enough signal to be worth weighting.

05

Idempotent async endpoints

Because the front end auto-retries past 150 seconds, a non-idempotent endpoint would double-fire under ordinary load. Idempotency is a requirement here, not a refinement.

How it was measured

A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.

Cross-tenant leakage under adversarial testingRun cost per tenant per monthLatency under the retry thresholdBehavior on duplicate delivery
figures

What it produces
Without this discipline

A per-tenant database that satisfies the security conversation, costs a multiple of what the feature earns, and hits a ceiling while the customer count is still small.

This system

Private per-customer similarity over an API with precomputed catalog embeddings, a configurable rerank stage for user-defined criteria, hard per-tenant isolation enforced below the app layer, and idempotent endpoints. Shipped with zero cross-tenant leaks under adversarial testing and run cost held to a few dollars per tenant per month.

one store, filtered below the appcriteria in rerankidempotent by requirementcost held per tenant
The operating envelope

What it owns, and what it hands to a person.

Handled with confidence
Per-tenant private catalogs
User-defined criteria at rerank
Client retries and duplicate delivery
Flagged for review
Tenants without enough signal for weighting
Out of scope by design
Cross-tenant similarity by design
Per-tenant model training
The honest limit

Isolation is enforced at the read path, which is the right place, and that makes the read path a component with no acceptable failure rate. The per-tenant weighting needs accumulated signal, so new tenants run on the global ranker until they have earned their own weights.

What it generalizes to

When a feature is priced per customer, the architecture is a cost decision before it is a capability decision. Put the expensive work where it happens once, put the variable work where it can change for free, and enforce the security boundary below the layer most likely to have a bug.

How we engage

You have a system like this one.
Tell us where it stands.

Whether it is failing, not yet built, or about to meet a scale it has never seen, we can tell you what we see.

Start a conversation
mostafa@opulion.dev · Response within 24 hours · By inquiry