The Number Someone Signs: Rebuild It by Rule, Do Not Read It Off the Document

Mostafa Dhouib·8 min read

The Number Someone Signs: Rebuild It by Rule, Do Not Read It Off the Document

The short answer. There is a class of automation where a professional personally certifies the output, and in that class the visible problem and the dangerous problem are different. The visible problem is extraction robustness across formats. The dangerous problem is that the total printed on the source document may include amounts that must not be claimed, because a rule outside the document excludes them. A pipeline that reads the stated total is correct about the document and wrong about the obligation. The figure has to be rebuilt from the line items by classification rule, and the system has to be scored against what was actually filed rather than against what was extracted.

Most document automation is scoped as an extraction problem, and for most document automation that is right. Get the fields out reliably across a hundred source formats, handle the scans, handle the rotated pages, measure field-level accuracy, ship.

There is a category where that framing produces something worse than useless, and the tell is simple: somewhere downstream, a named person puts their signature against a number your system produced, and carries personal responsibility for it being correct.

Perfect extraction, wrong number

Take a ledger. It has line items and a total. Your pipeline reads both flawlessly, every character correct, across every sender format in the corpus.

Now a rule outside that document says one category of charge on it cannot be claimed in this context. A fee that is enforceable under one agreement is prohibited under an addendum bound into a different one. The ledger does not know this. The ledger's total is arithmetically correct and includes the prohibited amount, because the ledger is a record of what was charged, not a calculation of what may be demanded.

Read the total and your extraction accuracy is 100 percent. The number is wrong. It is wrong in the specific direction of claiming more than is permitted, which in a regulated setting is the direction that carries consequences.

This is the shape of the problem. The document is a faithful record of one thing. The figure you need is a different thing, derived from the same line items under rules the document has no knowledge of.

Rebuild by rule

The architecture that follows is not complicated, and it is structurally different from extraction.

Extract the line items, not the total. The total is a derived value that belongs to the source system. Take the components.

Classify every charge. Each line item gets a category, and the categories are the ones the rule set cares about. In a tenancy context that is base rent against additional rent, because the forms distinguish them and the law treats them differently. In other domains it is reimbursable against non-reimbursable, in-period against out-of-period, covered against excluded.

Apply the exclusion rules, with their conditions. The rules are usually conditional on something outside the ledger: which agreement governs, which programme the party is enrolled in, which addendum is bound into the lease. Those conditions are inputs to the pipeline, not values to be read from the document being processed.

Sum what survives. That is the figure. It is a sum certain that can be defended line by line, which is exactly what a person certifying it needs.

Keep the derivation. Every number carries the line items it came from and the rule that admitted each one. When the figure is challenged, and in this class of work it will be, the answer is a trace rather than a recollection.

The document that overrides the agreement

The specific fact that makes this concrete, from a tenancy context: a federal tenancy addendum bound into a voucher lease overrides the lease's own fee clause. The lease says a fee is owed. The addendum, which is part of the same stack of paper, says it is not.

A pipeline reading the lease gets a correct answer from the lease and the wrong answer for the case. A pipeline reading the ledger gets a correct total and the wrong claim. The governing instrument is a third document, and knowing that it governs is domain knowledge that no amount of extraction quality supplies.

This is the general warning. In any regulated automation, ask which document governs, and whether the one you are reading is it. The answer is frequently no, and the one that governs is frequently the one nobody mentioned in the brief because everyone in the domain takes it for granted.

The exposure is not theoretical. A 2024 Seton Hall study found that 39 percent of subsidised-tenancy complaints sought prohibited fees. A 2026 Appellate Division opinion vacated a judgment on a subsidised lease because the sum certain of rent due was never stated. And under Hodges v. Sasil Corp., landlord-side law firms are debt collectors for the purposes of federal debt-collection law, so the consequences of an overstated demand are not merely procedural.

Score against what was filed

This changes the acceptance test, and it is the most transferable idea here.

Extraction accuracy measures your pipeline against the document. That is the wrong comparison, because the document is not the thing the pipeline is trying to get right.

Score against the filed artefact. If past matters exist with the final document attached, you have a set of cases where a qualified person already decided what the correct figure was. Run the pipeline on the same inputs and compare its output to what was actually sworn.

This gives you three things extraction accuracy does not. A ground truth produced by the accountable party rather than by an annotator. Coverage of the awkward cases in their real proportions. And disagreements that are genuinely informative, because every one is either a pipeline bug or a case where the human made a judgement your rules do not encode, and both are worth knowing.

The question to ask early in this kind of engagement is therefore not about document formats. It is whether past cases exist with the filed output attached, because that answer determines whether the system can be evaluated at all.

Where the rules stop

Rebuild-by-rule only covers the rules you have encoded. A case governed by a condition nobody stated is not handled correctly by this architecture, it is handled confidently and wrongly, which is the same failure as reading the total.

So the system has to have a third outcome. Not just a figure and an error, but a stop: this case involves a condition I do not have a rule for, and I am not producing a number. In work where a person certifies the output, an abstention is cheap and a confident wrong figure is not.

The honest limit

This describes an architecture, not a delivered result, and I am not claiming measured outcomes for it. The legal facts cited are specific to one jurisdiction and one programme, and they were current when researched. Anyone building in this area needs their own legal review rather than this article, which is about the engineering shape and not about the law.

The shape generalises well beyond tenancy: insurance claims, medical billing, expense reimbursement, regulatory filings, tax computation. Anywhere a person signs a figure a machine assembled, the same three moves apply. Rebuild from components rather than reading a stated total, find out which instrument actually governs, and score against what was filed rather than against what was extracted.

FAQ

Why is extraction accuracy the wrong metric for legal or financial automation? Because it measures the pipeline against the document, and the document is not what the pipeline is trying to get right. A ledger's total can be extracted perfectly and still include an amount that must not be claimed, because the rule excluding it lives outside the ledger. Score against the filed artefact instead, where a qualified person already decided the correct figure.

What does it mean to rebuild a figure by rule? Extract the line items rather than the stated total, classify each charge into the categories the rule set distinguishes, apply the exclusion rules with their conditions, and sum what survives. The result is a sum certain defensible line by line, with a stored derivation showing which items were admitted and under which rule.

How can a document be correctly extracted and still produce a wrong number? When a different document governs. A fee enforceable under a lease can be prohibited by an addendum bound into the same stack of paper, and the ledger recording the charge has no knowledge of either. Knowing which instrument governs is domain knowledge that extraction quality cannot supply.

What should an automated pipeline do with a case it has no rule for? Stop and say so. A figure and an error are not enough outcomes. There has to be a third that refuses to produce a number. Where a person certifies the output, an abstention costs a review and a confident wrong figure costs considerably more.

How do I build an acceptance set for this kind of system? From past cases with the final filed document attached. Those give you ground truth produced by the accountable party, the awkward cases in their real proportions, and disagreements that are each either a pipeline bug or an unencoded human judgement. Whether such cases exist in retrievable form is worth establishing before anything is built.

Does this apply outside legal work? Yes. Insurance claims, medical billing, expense reimbursement, regulatory filings and tax computation all have the same structure: a person signs a figure a machine assembled, and a rule outside the source document determines what may be included. The three moves are the same in all of them.