A false positive stops the line or consumes a maintenance window. A false negative ships bad product or lets a machine run to failure. Put your own numbers on both. If you have never priced them, that is the finding.
Move the threshold and watch the two error columns trade against each other. The steepness is how sharply your detector separates the two classes; if you have a real ROC curve, set it to match.
| Operating point | Selectivity | Recall | Specificity | Misses | False alarms | Annual cost |
|---|---|---|---|---|---|---|
| Your setting | 90.0% | 70% | 90% | 24 | 392 | $686,298 |
| Cost-optimal | 93.8% | 61% | 94% | 31 | 243 | $657,742 |
| Symmetric metric | 81.8% | 82% | 82% | 15 | 715 | $847,872 |
A system that alarms more often than anyone can act on gets overridden, then ignored, then switched off. That cost never appears in either error column, and the day it stopped being believed is never recorded anywhere.
Cost-optimal threshold, by cost ratio and prevalence
Where the operating point lands when you minimise expected cost. Read down a column to see what falling prevalence does, and across a row to see what happens as a miss gets more expensive than a stop. The pattern worth internalising: a rare condition forces selectivity up, and an expensive miss pulls it back down. At 2 percent prevalence a miss has to cost around 25 times a false alarm before the optimum comes off the top of the range.
| Miss cost vs stop cost | 0.5% prevalence | 1% prevalence | 2% prevalence | 5% prevalence | 10% prevalence |
|---|---|---|---|---|---|
| 1 to 1 | 99.9% | 99.9% | 99.9% | 99.4% | 98.3% |
| 2 to 1 | 99.9% | 99.9% | 99.6% | 98.4% | 95.9% |
| 5 to 1 | 99.8% | 99.5% | 98.5% | 95.0% | 89.1% |
| 10 to 1 | 99.5% | 98.5% | 96.3% | 89.6% | 80.2% |
| 25 to 1 | 98.0% | 95.2% | 89.9% | 77.5% | 63.4% |
| 100 to 1 | 90.1% | 81.6% | 69.6% | 49.6% | 33.6% |
Binormal ROC at AUC 0.90, a decent industrial detector. Selectivity is the percentile of clean cases the threshold sits above, so 99 percent means one false alarm per hundred clean items. Cyan is comparatively permissive, red is highly selective. The absolute numbers move with your detector; the direction of travel across the table does not.
A worked example
A vision inspection system judging 4,000 parts a year on a line where genuine defects run at 2 percent. A false positive stops the line and costs about 900 dollars in lost throughput and restart. An escaped defect reaches a customer and costs about 14,000 dollars. That is a ratio of roughly 16 to 1.
Nobody supplied those numbers when the system was tuned, so it shipped at the point that maximised a symmetric metric. That point weights a 900 dollar stop and a 14,000 dollar escape equally, which is the one assumption the plant would never have agreed to if anyone had put it in a sentence.
The fix is a day of work and it is not a model change. Ask the two people who own the line what a stop costs and what an escape costs, put the ratio into the threshold, and write both numbers down next to it in the repository, so that when somebody retunes it in eighteen months the reasoning is there rather than in a departed engineer's memory.
Then check part three. If the alarm rate at the new point produces more review than the team can absorb, the cost-optimal threshold on paper is not the one to ship, because a queue nobody drains is protection you are paying for and not receiving.
The arithmetic, so you can check it
Expected annual cost is misses x cost of a miss + false alarms x cost of a false alarm, where misses are volume x prevalence x (1 - recall) and false alarms are volume x (1 - prevalence) x (1 - specificity). The tool sweeps the threshold in one-percent steps and reports the minimum, which is a calculation you can reproduce in a spreadsheet in about ten minutes.
Recall and specificity here come from a stand-in detector rather than from your ROC curve, so the absolute thresholds are indicative. The decision this supports does not depend on that: it depends on the two costs being different, on somebody knowing what they are, and on the threshold being derived from them rather than inherited from a framework default. Substitute your measured curve and the method is unchanged.
The reasoning is in what a minute of line downtime buys you in architecture, and the trust failure in part three is the subject of human in the loop or safe state.
Questions
From the relative cost of the two errors at your site, not from the point that maximises accuracy or F1. Price a false positive and a false negative, then pick the threshold that minimises expected cost at your volume and prevalence. Those two numbers are a plant decision rather than a data science one, and they are usually available from the people who own the line.
Because both weight the two errors equally, which encodes the assumption that a stopped line and an escaped defect cost the same. When the real ratio is ten to one, tuning to a symmetric metric can leave a large amount of money on the table every year, and the gap widens as prevalence falls.
It rises, sometimes sharply. At low prevalence there are far more clean cases than real ones, so even a small false-positive rate produces many more false alarms than there are true cases to find. That pushes the optimum toward selectivity unless a miss is catastrophically more expensive than a stop.
It does not appear in either error column, and it is what kills these systems. A queue larger than the team can drain gets overridden, then ignored, then switched off, and an inspection or monitoring system nobody trusts carries cost without delivering protection. The day it stopped being believed is never recorded anywhere.
Yes. The detector quality control takes an AUC, which most teams either have or can estimate, and it drives a standard equal-variance binormal ROC. Set it to your measured AUC if you have one. The shape of the trade, and where the cost optimum sits relative to a symmetric choice, holds across reasonable settings; the point is that the threshold should come from your costs either way.