An extraction system pulling structured data out of messy real-world documents, sitting at 44% accuracy. Low enough that a person re-checked nearly everything, which defeats the automation entirely. A wrong extraction is expensive, and a model that is confidently wrong on the hard cases is more dangerous than one that flags them.
The stated ask was to make the model more accurate. The real problem is whether the system knows when it is unsure, so a person can be pointed at the doubtful cases instead of re-checking everything. Raw accuracy alone does not fix an automation that nobody trusts, because at any accuracy short of perfect the reviewer still has to look at all of it unless the system can say which ones to look at.
The reflex, and the fix.
Push raw accuracy
It is the stated ask, it is measurable, and it feels like the obvious engineering response.
Even a large accuracy gain leaves a reviewer checking everything, because without calibrated uncertainty there is no way to know which outputs to trust.
Make the system honest about its own doubt
Rebuilding the extraction path and adding calibrated uncertainty and a human review gate, rather than only tuning for a higher number.
A 95% system where the 5% it is unsure about is routed to a person, which is what makes the automation usable.
Fix the doubt, not just the accuracy, because the reviewer's time is the thing the automation was supposed to buy back.
The extraction path rebuilt
From 44% to 95% accuracy on messy real-world documents, in five days.
Output that carries calibrated uncertainty
The model returns a confidence that means something, rather than a bare answer that a reader has to take on faith.
A human review gate on the doubtful tail
Low-confidence cases are caught before they flow downstream. The gate is what turns the model is 95% into the 5% it is unsure about is routed to a person.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
44 to 95% in five days is a measured, delivered result.
A more accurate model that a reviewer still checks in full, because nothing tells them which outputs to distrust.
44 to 95% extraction accuracy in five days, with calibrated uncertainty and a human review gate on the low-confidence tail, so review effort lands where the doubt is.
What it owns, and what it hands to a person.
Accuracy is measured on the document mix this system actually receives. Calibration has to be maintained as the document population shifts, and the review gate assumes there is a person available to take the flagged cases.
This is the second clean proof of the thesis that it is almost never the model. The stated problem was accuracy, the real problem was self-knowledge, and the fix took five days against the multi-month rebuild that a model-first reading would have scoped. The discipline carries anywhere a human reviews machine output: make the system say what it does not know, and the review effort collapses onto the part that needs it.