Measure Precision Before You Set a Threshold

Mostafa Dhouib·10 min read

Measure Precision Before You Set a Threshold

The short answer. Take one sample, leave it exactly where it is, and photograph it five times in a row. Compute your derived value from each image. The five values will not be the same. That spread is your measurement precision, and it is a property of your instrument and method, not a mistake. Any decision threshold narrower than that spread produces a result that flips between readings of an unchanged sample, which is a coin toss with a number attached to make it look like a measurement. So precision gets measured first, in the terms the clinical precision standards already define, and the threshold is designed afterwards inside what the precision will support.

The order in which these two things happen determines whether the product works, and the order is almost always wrong.

What usually happens is that somebody picks a cut-off early, from a reference range, a paper, or a clinical collaborator's judgement. The software is built around it. Precision gets characterised much later, during validation, when the cut-off is already embedded in the user interface, the training material and the regulatory documentation. And at that point, if the precision turns out to be wider than the decision needs, there is no cheap fix available.

The five-photograph test

This takes ten minutes and it is the highest-value measurement in the whole project.

Set up a sample. Capture an image, compute the derived value, and record it. Without touching the sample, the device, the lighting, or anything else, capture again. Repeat until you have five.

Then look at the five numbers.

They will differ. There is no configuration in which they do not, because every stage of the chain contributes variation. Sensor read noise differs between exposures. Illumination fluctuates, including at the mains frequency. Automatic exposure, white balance and gain may re-converge differently. Compression quantises differently on near-identical frames. Segmentation picks a slightly different boundary, which changes which pixels are averaged. Background estimation lands in a slightly different place.

None of that is a defect. It is what measurement is. A camera-based assay is an instrument, and instruments have precision.

The important move is to treat the spread of those five values as the primary specification of the system, rather than as an annoyance to be reduced later. It tells you the smallest difference the system can distinguish, and therefore the finest decision it can honestly support.

Then do it again with the sample reseated between captures, because that difference matters enormously and is the subject of the next section.

The vocabulary already exists, and it is worth using properly

Clinical laboratory precision standards have a well-developed vocabulary for exactly this, and borrowing it saves a great deal of confused argument. The underlying metrological definitions come from the international standards on accuracy of measurement methods, and the laboratory precision experiment designs come from the clinical standards built on top of them.

The distinction that matters most:

Repeatability is variation under conditions held as constant as possible. Same sample, same operator, same device, same session, measurements close together in time. This is the five-photograph test with nothing touched. It is your noise floor, the best the system will ever do, and it is the number people quote because it is the flattering one.

Reproducibility is variation under deliberately changed conditions. Different operator, different device, different day, different lighting environment, different reagent lot, sample reseated. This is the number that governs whether a threshold works in the field, and it is always worse than repeatability.

Between them sit the intermediate conditions, which the clinical standards handle explicitly as components of a nested precision experiment. Within-run, between-run, between-day, between-operator, between-instrument. The point of the nesting is that it attributes variation to a source rather than reporting one lumped figure, and attribution is what tells you whether to spend on fixturing, on operator training, on calibration frequency or on the algorithm.

Two practical notes on using it.

Report precision at more than one concentration or level. Precision is rarely constant across the measuring range, and it usually behaves differently near the bottom of the range, which is frequently exactly where the clinically interesting threshold sits. A single precision figure quoted at a convenient mid-range level is not useful for a low cut-off.

Report the component estimates, not just the total. A total standard deviation tells you whether the decision is supportable. The breakdown tells you what to fix if it is not.

A threshold inside the noise is not a decision

Here is the mechanism, stated plainly.

Suppose your reproducibility spread is some width W, and your threshold sits at value T. A sample whose true value is within roughly W of T will land on either side of T depending on which measurement you happened to take. Run it twice and you can get two different answers. Hand it to a second operator on a second device and you can get a third.

From the user's point of view, the system is unreliable. From the system's point of view, it is working exactly as specified, reporting a measurement whose precision was known and a decision whose granularity was finer than that precision.

This is why threshold tuning is such an unsatisfying activity in these projects. Moving T changes which samples sit in the ambiguous band. It does not shrink the band. The width of the band is W, and W is a property of the instrument.

So the design sequence has to be:

Measure repeatability and reproducibility first, across the range, with the components attributed.

Ask what decision the clinical or operational use actually requires, and what difference it needs to distinguish.

Compare. If the required granularity is comfortably wider than the reproducibility spread, you can set a threshold and it will behave. If it is comparable or narrower, you cannot, and no software decision rule will rescue it.

If you cannot, change the instrument or change the output. Changing the instrument means controlled illumination, fixed geometry, an in-frame reference, locked camera parameters, replicate capture with averaging. Changing the output means not making the call.

Flag, do not call

This is the part that gets resisted, and it is the part that most often saves the product.

When precision will not support a binary decision, the correct output is not a decision with a disclaimer. It is a different output type.

Report the value with its uncertainty. A number with an interval is honest and is frequently more clinically useful than a category, because the person receiving it can combine it with context the device does not have.

Use three outcomes, not two. Clear positive, clear negative, and a band in the middle that is reported as indeterminate and routed to a confirmatory path. The width of the middle band comes directly from the measured reproducibility, which gives you a principled way to set it rather than an argued one.

Require replicates near the boundary. If a single reading lands in the ambiguous band, capture more and report the mean with its interval. This is a direct and cheap use of the precision data, since the uncertainty of a mean shrinks with the number of replicates in a known way.

Make refusal a first-class result. If the image fails quality checks, if the in-frame reference is out of range, if the lighting has drifted, the device should say so and decline rather than produce a value nobody can qualify. A declined measurement is a good measurement. A confidently wrong one is the expensive failure.

The organisational resistance to this is predictable, because an indeterminate band looks like a worse product on a specification sheet. It is a better product in use, and it is far easier to defend when somebody asks how the threshold was chosen.

The honest limit

Precision is not accuracy, and measuring one tells you nothing about the other. A system can be beautifully repeatable and consistently wrong, because a systematic bias affects every reading identically and therefore does not appear in any precision experiment. Trueness needs reference materials and a comparison against a method of known accuracy, which is separate work with a separate cost.

A precision study also characterises the conditions it sampled. A reproducibility estimate from two operators on two devices in one building is not a reproducibility estimate for deployment across many sites with uncontrolled lighting. The nested design tells you how much variation came from each source you varied, and nothing at all about sources you did not vary.

And the five-photograph test is a screening instrument, not a precision study. Five captures with nothing touched gives you a quick, honest look at the repeatability floor, which is usually enough to kill an unsupportable threshold early. The formal nested experiment the clinical standards specify is what you need for a validated claim, and it is considerably more work. The point of the quick version is to run it before the cut-off is designed in, when the answer can still change the design.

FAQ

Why do five photographs of the same sample give different values? Because every stage of the chain contributes variation. Sensor read noise differs between exposures, illumination fluctuates, automatic exposure and white balance can re-converge, compression quantises near-identical frames differently, and segmentation picks a slightly different boundary so a different set of pixels is averaged. That spread is the instrument's repeatability, and it is the floor on what the system can distinguish.

What is the difference between repeatability and reproducibility? Repeatability is variation with conditions held as constant as possible: same sample, operator, device and session. Reproducibility is variation under deliberately changed conditions: different operator, device, day, lighting or reagent lot, with the sample reseated. Repeatability is the flattering number and reproducibility is the one that governs whether a threshold survives in the field.

Can I fix an unreliable result by moving the threshold? No. Moving the cut-off changes which samples fall into the ambiguous band but does not change the width of that band, because the width is set by the measurement precision rather than by the decision rule. A sample whose true value sits within the reproducibility spread of the threshold will flip between readings regardless of where the threshold is placed.

What should the device output when precision will not support a call? Three outcomes rather than two, with the middle band sized from the measured reproducibility and routed to a confirmatory path. Alongside that, report the value with an uncertainty interval, require replicates near the boundary, and allow the device to decline outright when image quality or the in-frame reference falls out of range. A declined measurement is a good measurement.

Does a good precision result mean the assay is accurate? No. Precision and trueness are independent properties. A systematic bias shifts every reading by the same amount, so it never appears in a precision experiment, and a system can be highly repeatable while being consistently wrong. Establishing accuracy requires reference materials and comparison against a method of known trueness, which is separate work.

When in the project should precision be measured? Before the threshold is designed, which is almost always earlier than it happens. Once a cut-off is embedded in the interface, the training material and the regulatory file, discovering that the precision cannot support it leaves no cheap options. The quick five-capture repeatability check takes minutes and can be run on the first working prototype, long before a formal nested precision study is feasible.