· 10 min read
A Confidence Score That Never Changes Cannot Route Your Review Queue
Calibration in aggregate and the ability to rank individual items are two different properties, and a score can have the first without the second. In a lab reproduction, a calibrated but nearly constant confidence score caught 4 percent of errors at a 5 percent review budget, against 95 percent for an informative score on the same data. Threshold tuning cannot repair a score that does not rank, because there is no threshold at which a flat score separates anything.