The Verifier That Verifies Nothing

Mostafa Dhouib·September 5, 2026·8 min read

The Verifier That Verifies Nothing

The short answer. A component named verifier is not a verifier. If it is a model call asking whether the output looks correct, it produces an opinion, it shares the generator's blind spots, and it has probably never rejected anything. A real verifier has three properties: it is deterministic, it is independent of the thing it checks, and it is grounded in something outside the system. There is a five-rung ladder from opinion to execution, and the honest first step is finding out which rung you are on.

Open almost any production AI pipeline and you find a stage called verifier, or validator, or critic, or reviewer. Its presence on the architecture diagram is doing a lot of work in the room where budgets are approved.

Then you look at what it does, and it is a second call to the same model with a prompt that ends "is this correct? Answer YES or NO."

Naming a component after the property you want

This is the general failure and it is worth naming, because once you have the name you see it everywhere in AI architecture.

A component called guardrail does not restrain anything by virtue of its name. A layer called safety is not safe. A stage called verifier does not verify. The name records an intention; the code either has the property or it does not, and the diagram cannot tell you which.

The reason this survives review is that the diagram is what gets reviewed. Everyone in the room can see that there is a verification step, and the question that would settle it is not asked: what has this rejected?

The three properties a real verifier has

If you want to know whether a stage is doing verification, check it against three things. Each of them is a yes or no question and none of them requires reading much code.

It is deterministic. Given the same input it returns the same verdict, always. A sampled model opinion is not a function of its input; run it twice on identical text and you can get two answers. This is the same defect that makes a model's "I think I am done" unusable as a stopping condition: it is not a function of state.

It is independent of what it is checking. Verification only adds information if its errors are uncorrelated with the generator's. The same model, with the same training and the same priors, asked about its own output, is not an independent observer. It made the mistake because of what it believes; asking it whether it made a mistake consults the same beliefs. Where the generator is confidently wrong, the checker is confidently agreeable.

It is grounded in something outside the system. A verdict has to be traceable to a fact the system did not generate. Does it compile. Do the tests pass. Does the JSON parse against the schema. Does the total reconcile with the ledger. Does the row exist in the database. Absent that, the pipeline is comparing one of its own outputs against another of its own outputs.

Independence is the load-bearing one, and it is the one nobody checks, which is why this shows up under a different name in the retrieval world: AI cannot grade its own homework is the same structure applied to evaluation.

The ladder

Verifiers are not binary. There are rungs, they cost different amounts, and knowing which one you are on is most of the value.

Rung one: same model, asked for an opinion. Adds latency, adds cost, adds a box to the diagram. Correlated with the generator on exactly the cases you care about. This is where most production systems actually sit.

Rung two: a different model, asked for an opinion. Better, because the errors are less correlated. Still not deterministic, still not grounded, and the independence is partial: models trained on overlapping data share more priors than the vendor names suggest.

Rung three: a deterministic check on form. Schema validation, type checks, required fields present, numeric ranges, enum membership. Deterministic and cheap. It catches malformed output and it says nothing about whether the content is right, which is a real limit but a clearly stated one.

Rung four: a deterministic check against ground truth the system did not produce. The referenced invoice exists and its total matches. The cited passage appears in the retrieved document. The extracted date falls inside the contract term. This is where verification starts being worth the name.

Rung five: execution against reality. Run the code. Apply the migration to a copy and diff it. Submit the transaction to a sandbox. Reality is the only checker with no shared priors at all.

Every rung above three requires you to have something to check against, and that is the actual work. It is why teams stay on rung one: rung one requires no domain modelling, and it looks the same on the diagram.

Where verification is cheap, and where it is not

The reason this matters more for some systems than others is an asymmetry.

For some tasks, checking an answer is dramatically cheaper than producing it. Does the code compile. Do the tests pass. Does the sum reconcile. Does the extracted field appear verbatim in the source document. In these cases a rung-four or rung-five verifier is available, cheap, and the single highest-leverage thing you can add to the pipeline.

For other tasks, checking is as hard as producing. Is this summary faithful. Is this legal analysis sound. Is this the right strategic recommendation. Here there is no cheap grounded check, and dropping in a model-based verifier does not create one. It creates a rung-one component and a belief that the problem is handled.

The honest architecture for the second case is different: you do not add a fake verifier, you add a human review gate on the subset that matters, and you spend the engineering effort on making that subset small and well-chosen. That is designing for irreversible actions rather than pretending the check exists.

The question to ask on Monday

One question settles it, and it takes an afternoon to answer:

How many times has the verifier rejected something in the last thirty days, and what were those cases?

Three outcomes, all of them informative.

Nobody is counting. Then the component has never been evaluated and you have no evidence it does anything. Start counting; that instrumentation alone is often the whole finding, and it is the same reason every metric is a proxy for the thing you actually care about.

It has never rejected anything. Either the generator is perfect, or the verifier is a rubber stamp. Construct a case you know is wrong, feed it through, and see what happens. If it passes, you have your answer.

It rejects at some rate. Now sample the rejections and check them by hand. A verifier that rejects the wrong things is worse than none, because it adds cost and retries while filtering out good output.

The one sentence

A verifier is defined by what it can reject, not by what it is called. If you cannot describe an input that would make it say no, and demonstrate that it does, it is a stage that adds latency to a pipeline and confidence to a meeting.

FAQ

What makes something a real verifier? Three properties: it is deterministic, so the same input always yields the same verdict; it is independent of the component it checks, so its errors are not correlated with the generator's; and it is grounded in a fact the system did not itself produce. A model call asking "does this look right" has none of the three.

Why can't a model check its own output? Because it made the mistake for a reason, and asking it about the output consults the same beliefs that produced the error. Verification only adds information when the checker's errors are uncorrelated with the generator's, and the same model on the same input is maximally correlated.

Is using a different model as a verifier good enough? It is better than using the same one, because the errors are less correlated. It is still nondeterministic and still ungrounded, and models trained on overlapping data share more priors than their vendor names imply. Treat it as rung two of five, not as verification.

What if my task has no cheap way to check the answer? Then say so rather than adding a component that pretends otherwise. The honest design is a human review gate on the subset where being wrong is expensive, with engineering effort spent on making that subset small and well chosen.

How do I find out whether my verifier does anything? Count its rejections over the last thirty days. If nobody is counting, that is the finding. If it has never rejected anything, construct a case you know is wrong and feed it through. If it does reject, hand-check a sample, because a verifier that rejects the wrong things is worse than not having one.