Why You Can't Let an AI Grade Its Own Homework

Mostafa Dhouib·September 4, 2026·7 min read

Why You Can't Let an AI Grade Its Own Homework

The short answer. The single most important rule in evaluating an AI system: the check cannot be the same thing it is checking. Use one model to write the answer and the same model to judge it, and the loop closes on itself, keeping the identical blind spots twice. It also goes easy on itself in ways that are measurable rather than metaphorical: it prefers longer answers and its own writing style and scores those higher. The sharpest question you can ask anyone building your evaluation is what is doing the checking, and if the answer is any version of the AI checks itself, you have found your problem.

Every evaluation system answers one question: is this output right? Everything depends on what you appoint to answer it.

The convenient answer is to ask a model, because it is cheap, fast, and scales to forty thousand answers in a way a human does not. The convenient answer is also frequently the same model that produced the output, or a sibling of it, and that is where it stops working.

The loop that closes on itself

If the writer and the grader are the same model, then whatever the model does not know, it also does not know it does not know. Whatever shortcut it took while writing, it will take again while grading, because it is the same geometry making both decisions.

A blind spot is not a random error. It is a systematic region where the model's behaviour is wrong in a consistent direction. Asking the same model to check its own output samples that region twice and gets the same answer both times, then reports the agreement as confidence.

This is the student grading their own exam. The result is not random noise, it is an A.

It is measurable, not figurative

The important part, and the reason this is worth stating as a rule rather than as a caution, is that self-preference is not a hand-wave about vibes. It shows up as concrete, reproducible bias.

A model grading its own output prefers longer answers. It prefers its own writing style and phrasing conventions. It rates outputs that resemble what it would have produced higher than outputs that do not, independent of whether they are correct.

Which means a self-graded evaluation is not merely weak, it is actively pointed in a direction. Optimise against it and you will reliably produce answers that are longer and more stylistically self-similar, and you will watch your score climb while nothing improves. That is Goodhart's law with the model supplying both the metric and the gradient.

What a real check looks like

Three layers, and they are not alternatives. The value comes from combining them, because each catches what the others miss.

A golden set. Questions you already know the correct answer to, verified by someone who knows the domain, with the source recorded. This is the answer key, and it is the cheapest of the three. Run it whenever anyone changes anything. In the case that started this line of work, this alone would have caught the problem on day one.

It has to be maintained as a living thing. A golden set written eighteen months ago and never revisited will happily confirm that your system still gives the answer that was correct eighteen months ago. When the world changes, the answer key changes with it, which means the process that changes a policy is the process that touches the set.

Shadow and canary. Test a change quietly against real traffic without showing anyone the results, then release it to a small fraction of users with an automatic rollback. This is what stops you flipping a change on for everyone and hoping.

A human on a sample. One or two answers in a hundred, read by a person who checks them against the actual sources. This is the most expensive layer, the first one cut when a schedule tightens, and the one that matters most, because it is the only place real truth enters the system at all. Everything else is a proxy checked against another proxy.

An independent model grader has a place inside this, and it is a real one: it scales, it is cheap, and it catches obvious failures. It is a fourth layer, not a replacement for the three, and the word doing the work is independent.

What independent actually requires

"Use a different model" is the usual advice and it is necessary rather than sufficient. Three properties matter more than the model identity.

It should not share the failure mode. Two models trained on similar data with similar objectives share blind spots even with different weights. Independence in the sense that matters is about mechanism, not vendor. A deterministic check, a retrieval-based verification against the source, or a rule you can compute shares nothing with the generator, which is why those are stronger than a second model even when they cover less.

It should not see the generator's reasoning. Handing the grader the writer's chain of thought contaminates the check. It is now evaluating an argument rather than an answer, and a persuasive wrong argument is exactly the case you were trying to catch.

It should have access to the ground truth, not just the output. A grader that only sees the answer is judging plausibility. A grader that sees the answer and the source document can check correspondence, which is a different and much harder-to-fool question.

The hiring question

There is one question that tells you more than any resume, and it is short enough to ask in a meeting.

How do you check that the answers are right, and what is doing the checking?

If the answer is any version of the AI checks itself, you have found the problem and it is standing in front of you.

A good answer has three things in it: an answer key, a human looking at samples, and a grader that is not the thing being graded. If someone offers you fully automatic certainty with no humans anywhere, they are overselling. The honest version always has a person in the room and is specific about where its blind spots are.

Which leads to the follow-up that beats any pitch: what does your approach miss? Someone who really does this has a quick, specific answer, because they live with the limit every day. Someone overselling says it misses nothing, which is always false.

The general rule

This is not a retrieval rule or an AI rule. It is the reason auditors are not employees of the company they audit, why a test that has never failed proves nothing, and why a driver cannot be the thing that confirms its own transfer completed.

A check that shares a mechanism with the thing it is checking is not a check. It is the same computation, run twice, reported as agreement.

FAQ

Can an LLM evaluate its own output? Not reliably. It keeps the same blind spots in both roles, because the same geometry makes both decisions, and it exhibits measurable self-preference: it favours longer answers and its own writing style and scores those higher regardless of correctness.

What is LLM-as-a-judge bias? Systematic preference by a model grading outputs, most notably for length and for its own stylistic conventions. It matters because optimising against a biased grader reliably produces longer, more self-similar answers while the score climbs and nothing improves.

Is using a different model enough to make the check independent? It is necessary but not sufficient. Models trained on similar data with similar objectives share blind spots regardless of weights. Independence that matters is about mechanism, so a deterministic rule or a verification against the source document is stronger than a second model even when it covers less.

How should I evaluate a RAG system properly? Three layers combined: a maintained golden set as the answer key, shadow and canary releases against real traffic with automatic rollback, and a human reading a small continuous sample against the sources. An independent model grader is a useful fourth layer, not a replacement for the three.

What should I ask someone building my AI evaluation? How do you check that the answers are right, and what is doing the checking. A good answer contains an answer key, a human on samples, and a grader that is not the thing being graded. Then ask what their approach misses; a real practitioner has a quick specific answer.