Document PipelineField engagement

A document pipeline lives or dies on the documents it actually receives, not the clean ones the benchmark ships with.

Fractional CTO and lead engineer.
96.8%
accuracy
50,000+
documents a month
~50
parts, not 11,000 blocks
initial
medial
final
ONE LINE, FOUR SCRIPTS
한국어Hangul
漢字Hanja
A1Latin
2024digits
Hangul is composed, not enumerated, and one line can mix four scripts. Decode the parts and route the scripts apart.
What was at stake

Everything downstream inherits the recognizer's errors. A missed character is a missed search hit, a wrong digit is wrong data, a wrong field is a wrong decision, and no model downstream recovers a character the recognizer never read. Its accuracy sets the ceiling for everything built on top.

The constraint

The target script is composed, not enumerated. The unit on the page is a syllable block built from parts, and a single line can interleave multiple scripts. That makes the modeling-unit choice decisive: compose from roughly fifty parts, or enumerate on the order of eleven thousand blocks, and that decision determines whether the labeled-data requirement is tractable or a fortune. Most failures are also decided before recognition runs, in preprocessing, on real scans that are skewed, warped, low-DPI, and degraded.

Syllable blocks, learned directly~11,172
Jamo parts, composed~51
Model the fifty parts, not the eleven thousand wholes. The same decision separates a recognizer that needs a fortune in labels from one that does not.
The fork

The reflex, and the fix.

Road not taken

Tune an off-the-shelf, English-centric engine

Pull

Running in an afternoon, and fine on clean typed samples.

Why not

No model for a composed script, mixed-script lines, or real-scan degradation. It fails exactly where the documents live.

Road taken

Build for the script and the scans as they are

Accepted

Composition-aware recognition, mixed-script routing, a real-document evaluation set, and a synthetic-data strategy.

Bought

Accuracy on the documents actually received, at production volume.

Decision

Model the structure of the script and meet the scan where it is, because both are decided before recognition runs.

How it was built
01Preprocess
02Layout
03Recognize
04Post-process
05Evaluate
01

Meet the scan where it is

Deskew, dewarp, denoise, normalize DPI, and binarize until the page is clean, level, and legible. Most OCR failures are decided here, before recognition runs at all.

02

Structure before text

Layout analysis recovers regions, reading order across columns, and table structure, so the output is meaningful rather than a correct but shuffled stream of characters.

03

The modeling-unit decision

Decoding the script as composed parts keeps the label space small and the model data-efficient, against enumerating thousands of whole blocks. Scripts are routed apart so a character from one is never confused for a block from another, which is the most common mixed-script error.

04

Manufacture the data reality withholds

Labeled scans in this script are scarce, so training pairs are synthesized from real corpora across many fonts, then degraded to look like the actual archive: low DPI, bleed, stamps, fax noise. The closer the synthetic degradation matches, the more the accuracy transfers.

How it was measured

A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.

Character error rate per scriptCharacter error rate per document qualityReview rateThroughput at production volume
figures

What it produces
Without this discipline

Blocks dropped or split into wrong syllables, scripts confused for one another, tables flattened into a shuffled list, and one opaque output with no signal about what to trust.

This system

96.8% accuracy at 50,000+ documents a month, with composed syllables correct, scripts kept apart, tables preserved, reading order intact, and a confidence on every field so a reviewer checks only what needs it.

composition-awaremixed-script routingsynthetic degradationper-script evaluation
The operating envelope

What it owns, and what it hands to a person.

Handled with confidence
Printed text across the supported scripts
Tables and columns
Skew, noise, low DPI, common scan degradation
Flagged for review
Low-confidence fields
Ambiguous mixed runs
Out of scope by design
Free handwriting
Severe damage
Wholly novel layouts
The honest limit

It is strong on printed text across scripts and degradation levels. Handwriting, severe damage, and wholly novel layouts are still hard, which is exactly why low-confidence output is flagged for a human rather than asserted.

What it generalizes to

A document pipeline is won or lost on the documents you actually receive. Model the structure of the script, route what differs, manufacture the data reality withholds, and measure where it actually fails rather than where the benchmark does.

How we engage

You have a system like this one.
Tell us where it stands.

Whether it is failing, not yet built, or about to meet a scale it has never seen, we can tell you what we see.

Start a conversation
mostafa@opulion.dev · Response within 24 hours · By inquiry