Tagged model
5 pieces
- · 8 min read
How to Build a Test Set That Actually Predicts Production
A test set drawn at random from your training data measures how much your test set resembles your training set, not whether the system works. Building one that predicts production means holding out whole groups rather than random samples, choosing the grouping variable that matches how your system will actually be deployed, sizing the held-out groups honestly, and checking for the leakage that inflates a score while nothing was learned.
- · 15 min read
Why Your AI Works in Testing but Fails in Production
When a model scores 95% in staging and collapses to 44% in production, the model is almost never the cause. The usual fix, rebuilding the model, retrains on the same data and reproduces the same failure. The real problem is what the model learned from the data, and it is usually fixable in days, not months.
- · 11 min read
Your AI Is Memory-Bound, Not Compute-Bound (So Tuning the Model Won't Fix Latency)
On fixed hardware, inference latency is set by moving bytes, not by doing math. Most models sit in the memory-bound region with the compute units idle, so the real cost is moving weights and activations across a fixed memory bandwidth. Accuracy, latency, memory, and power are not four goals, they are four views of one thing, the bus, and the worst-case tail is set by contention, not by your model.
- · 12 min read
Confidence Is Not Evidence: Why an AI Model Can't Tell You When to Trust It
A model is geometry fit to data, so past the edge of its training it does not stop, it extrapolates, applying a flat rule that was anchored to nothing, with full confidence. Its certainty is not evidence, because standard training optimizes knowing the answer and never optimizes knowing whether it knows. The decision to trust it has to live outside the model.
- · 9 min read
Cleaning Your Input Can Make Your AI Worse
A model does not read your input, it reads a compressed representation of it, keeping only the features it found useful and throwing the rest away. So cleaning up a messy input with a denoiser polishes exactly the information the model already discarded, and moves the features it actually reads in a direction you cannot see, which is why input quality can go up while accuracy goes down.