Tagged diagnose
6 pieces
- · 7 min read
The Delete-the-Variable Test
A ten-second check that tells you whether a variable is doing real work or is scaffolding: delete it, rerun, and see what changes. Delete TCP's congestion window and the internet falls over, which it did in 1986. Delete your agent's iteration cap and the trace does not flinch. The test has one important limit, and knowing it is what stops the test becoming a licence to remove your own safety nets.
- · 8 min read
Rebuild or Repair? How to Tell Before You Approve Six Months
Two contractors quoted four and six months to rebuild a model. The actual fix took five days and was not the model. The decision between rebuilding and repairing is not a judgement call, it is a diagnosis you can run in an afternoon, and running it first is the difference between a five-day fix and six figures spent reproducing the same failure.
- · 7 min read
The Five Ways a Loop Fails to Stop
Floyd gave us the conditions a terminating loop has to satisfy in 1967. Negate them one at a time and you get an exhaustive list of the ways termination can fail, which turns a vague symptom into five named diagnoses with five different fixes. Your agent is almost certainly number four, and number four is the only one that looks like success.
- · 7 min read
Every Metric Is a Proxy: Goodhart's Law for AI Systems
You cannot read forty thousand answers, so you appoint one number as a messenger and send it to check the truth. Every dashboard number is that: a stand-in for something you actually care about but cannot watch directly. The gap between the number and the thing is where trouble hides, and the moment the number becomes the goal, people find ways to move it that have nothing to do with what you wanted.
- · 8 min read
Before You Blame the Model: The Four Boring Checks
When an AI system works in testing and fails in production, the model is the last thing to suspect, not the first. Four checks rule out the plumbing before you touch anything interesting: is the same model actually running, is it getting the same input, is the preprocessing identical, and is something quietly timing out or falling back. Most reported model failures die at one of these four.
- · 15 min read
Why Your AI Works in Testing but Fails in Production
When a model scores 95% in staging and collapses to 44% in production, the model is almost never the cause. The usual fix, rebuilding the model, retrains on the same data and reproduces the same failure. The real problem is what the model learned from the data, and it is usually fixable in days, not months.