Tagged data-quality
8 pieces
- · 8 min read
Schema-Valid and Ten Thousand Times Wrong: Units Are Semantics, Not Syntax
A strict JSON schema on an extraction pipeline validates type, range and required fields. It cannot see that the number 12 in a table headed in ten-thousands means 120,000. The row passes every check, lands in the database as 12, and is wrong by four orders of magnitude with no validation error anywhere. The fix is an arithmetic cross-check against a total the document already states.
- · 8 min read
The Number Someone Signs: Rebuild It by Rule, Do Not Read It Off the Document
When a person certifies a figure your system produced, extraction accuracy is the wrong metric. The document your pipeline reads can be perfectly extracted and still contain a total that must not be claimed, because a rule outside the document excludes part of it. The deliverable is a figure rebuilt by classification from the line items, with each charge categorised, and scored against what was actually filed rather than against what was extracted.
- · 8 min read
Your Decimal Engine Is Still Eating Floats: Determinism Is Decided at the Parser Boundary
Switching a calculation engine to decimal arithmetic fixes nothing if the numbers arrive as floats. A spreadsheet stores numeric cells as IEEE-754 doubles, the usual dataframe library infers a float type on read, and constructing a decimal from a float inherits the float's error exactly. The engine then computes precisely on a value that was already wrong, and nothing in the stack will tell you.
- · 7 min read
Your Dates Are 543 Years Off, and Every Row Loaded Successfully
A calendar convention read as the wrong system shifts every date by a fixed offset that can be centuries, and nothing rejects the result because the output is a valid date. The same class covers text moving through a legacy code page, numeric separators, and time zones stored without one. These failures pass row counts, null checks and schema validation, which is why a migration has to test for them explicitly rather than wait for them.
- · 7 min read
You Cannot Deduplicate Before You Normalise: Identity Is a Decision, Not a Property
Two records from different sources cannot be compared until they share a shape, so deduplication is downstream of normalisation whether or not anyone planned it that way. The harder half is that identity then has to be defined deliberately: nothing in the data says which fields make two records the same thing, and that choice decides both your duplicate rate and your merge errors.
- · 10 min read
Restaurant AI Fails on Three Numbers
Sales, labour and cost of goods each mean something specific in food and beverage, and a generalist gets all three wrong in directions that look plausible. Sales taken gross rather than net of comps, voids and discounts. Labour taken as wages rather than burdened with taxes and benefits. Cost of goods recognised from a stock count rather than from consumption, so product sitting in the walk-in is treated as sold. If the figures do not tie back to the source systems, no insight should be produced.
- · 11 min read
Withhold the Insight When It Does Not Reconcile
An operator reading a figure cannot tell a reconciled number from an unreconciled one, so a plausible narrative over numbers that do not tie back is indistinguishable from a correct one. Silence is the correct output. Reconciliation therefore has to be a gate in the pipeline rather than a report consulted afterwards, which means designing what the product shows when it shows nothing.
- · 7 min read
Attribute at Capture, Never Infer Later: An Untagged Record Belongs in a Bucket, Not in an Estimate
Attribution reconstructed after the fact from session data, timing or traffic patterns is a model presented as a measurement, and it is confidently wrong in the cases that matter most. Capture the context at the moment the thing happens, store it on the record, and put what arrives without context into an explicit unattributed bucket rather than apportioning it.