Schema-Valid and Ten Thousand Times Wrong: Units Are Semantics, Not Syntax

Mostafa Dhouib·8 min read

Schema-Valid and Ten Thousand Times Wrong: Units Are Semantics, Not Syntax

The short answer. Schema validation checks shape. It confirms a field is a number, that it is present, and that it falls in a range. It cannot confirm the number means what the pipeline assumes, and the most expensive version of that gap is units. A Chinese financial table headed 万元 reports amounts in ten-thousands, so a cell reading 12 means 120,000. Extract it as 12, validate it against a strict schema, and every check passes: it is a number, it is present, it is positive, it is plausibly sized. The value is wrong by a factor of ten thousand and nothing in the pipeline has any way to know.

Strict output schemas have become the standard defence for structured extraction, and they earn their place. They eliminate a real category of failure: missing fields, wrong types, hallucinated keys, text where a number belongs. Constrained decoding makes that category close to impossible.

The confidence this creates is the problem. A pipeline whose outputs always validate feels like a pipeline whose outputs are correct, and those are different claims. Validation tells you the shape is right. It is silent on meaning.

The 万元 case

In Chinese financial documents, amounts are very often presented in units of 万元, ten-thousand yuan. The unit is declared once, in the column header or a note above the table, and the cells below carry bare numbers.

So a revenue line showing 12 means 120,000 yuan.

An extraction pipeline that reads the cell and not the header produces {"amount": 12}. Run it through the strictest schema you like. It is a number. It is required and present. It is positive. If you have a range constraint, 12 sits comfortably inside any range that also has to accommodate the rest of the document.

Every check passes. The value is off by four orders of magnitude.

This is worse than a parse failure in a specific way: a parse failure stops the pipeline and somebody looks at it. This produces a clean row that flows into a database, into a report, into a comparison against other figures that may or may not have the same problem, and the first signal is that a total somewhere does not make sense, weeks later, with no trail back to the cause.

Why the model does not save you

The language model reading the document can often see the header. Sometimes it applies the conversion correctly. The trouble is the "sometimes".

Whether it converts depends on whether the header was in the chunk, how the prompt was worded, and how the table was serialised when it was passed in. A pipeline that converts correctly on documents where the header is near the number and incorrectly on documents where it is three pages earlier is a pipeline with a four-order-of-magnitude error correlated with document layout, which is close to the worst possible error distribution. It will be right on your test set, which was assembled from clean documents, and wrong on the archive.

You also cannot tell which happened by looking at the output. There is no flag that distinguishes "the model converted" from "the model copied the cell". Both produce a number.

The fix is arithmetic, not stricter schemas

A stricter schema cannot close this, because the information required is not in the field. It is in the relationship between the field and the document.

Two mechanisms do close it.

Make the unit an explicit extracted field. Do not let the pipeline convert silently. Extract the amount and the unit as separate values, with the unit required. Then conversion is a step in your code, visible and testable, rather than a judgement the model may or may not have made. A document where the unit cannot be determined fails loudly, which is the correct outcome, because the alternative is a number nobody can trust.

Cross-check against a total the document already states. Financial tables almost always carry a total row, a subtotal, or a figure repeated in prose. Sum the extracted line items and compare. A unit error on one column makes the sum wrong by a factor of ten thousand, which is not a near miss, it is the loudest possible signal. This catches unit errors, missed rows, duplicated rows and sign errors with a single check, and it costs one comparison.

The second is the one that generalises. Documents of record are usually internally redundant, because humans need to check them too, and that redundancy is a free validator sitting unused in most extraction pipelines.

The confidence score that catches nothing

There is a related trap worth naming, because it is the thing people reach for next.

If you route uncertain extractions to a reviewer, the score doing the routing has to be informative, meaning it is higher on correct outputs than on incorrect ones. A score that is well-calibrated in aggregate but nearly constant across items is not informative, and it is common, because models are often confident in a flat, uniform way.

In the reproduction I ran, a calibrated-but-constant confidence score caught 4 percent of errors at a 5 percent review budget. An informative score at the same budget caught 95 percent.

Same budget, same reviewer, same number of items looked at. The difference is entirely whether the score ranks errors above correct outputs, and a score that is constant ranks nothing. You are sampling at random and paying for a review process that is approximately decorative.

Test this directly: sort by confidence, take the lowest 5 percent, and measure what share of your known errors you captured. If it is near 5 percent, your score is not ranking, and no threshold tuning will fix it.

What to check in your own pipeline

Find every field whose meaning depends on something outside the field: units, currencies, scales, date formats, fiscal versus calendar periods, inclusive versus exclusive ranges. Each one is a place where a schema-valid value can be semantically wrong.

For each, decide whether the context is extracted explicitly or assumed. Assumed is the failure.

Then find the redundancy the document already contains, and use it. A total, a count, a checksum, a figure stated twice in different places. That is your validator and it is more reliable than any schema.

The honest limit

The 4 percent and 95 percent figures are from my own reproduction, constructed to isolate the effect of an informative score against a constant one. They are a demonstration of the mechanism rather than a measurement of any particular production system, and the numbers you get will depend on your error rate and how your score behaves.

The 万元 example is one instance of a general class, and I want to be careful not to overstate the specific case: a competent pipeline on clean, well-formed documents with the header adjacent to the table often handles it correctly. The failure is concentrated in the awkward documents, which is both why it survives testing and why it matters.

FAQ

Can a strict JSON schema catch a unit error in extracted data? No. A schema validates shape: type, presence, range. A cell reading 12 under a header declaring units of ten-thousands is a valid number, present, positive and within any plausible range, so every check passes while the value is wrong by four orders of magnitude. The information needed to catch it lives in the relationship between the field and the document, not in the field.

Why is a unit error worse than a parse failure? A parse failure stops the pipeline and somebody looks at it. A unit error produces a clean, valid row that flows into the database, the report and every downstream comparison. The first signal is usually a total that does not make sense weeks later, with no trail back to the cause.

How do I stop an extraction pipeline from getting units wrong? Extract the unit as its own required field rather than letting the model convert silently, so conversion becomes a visible step in your code and an undeterminable unit fails loudly. Then cross-check the sum of line items against a total the document already states, which catches unit errors, missed rows and duplicates in one comparison.

Why doesn't the language model just apply the conversion? Sometimes it does. Whether it does depends on whether the unit header was in the same chunk, how the table was serialised, and how the prompt was worded, which means the error correlates with document layout. It will be right on a clean test set and wrong on the awkward documents in the archive, and the output gives no indication which happened.

Why does my confidence-based review process catch so few errors? Probably because the score is calibrated but nearly constant, so it does not rank errors above correct outputs. In reproduction, a constant score caught 4 percent of errors at a 5 percent review budget against 95 percent for an informative one. Sort by confidence, take the lowest 5 percent, and measure the share of known errors captured. Near 5 percent means you are sampling at random.

What makes a good validator for document extraction? The redundancy already in the document. Documents of record carry totals, subtotals, counts and figures repeated in prose, because humans need to check them too. Summing extracted line items against a stated total is a stronger correctness check than any schema, and most pipelines leave it unused.