Tagged reliability
36 pieces
- · 7 min read
Your Agent Framework Was Specified in 1990 (And Is Missing the Same State)
In 1990 Cohen and Levesque formally specified what it means for an agent to hold a goal: it believes the goal is not yet achieved, it wants it achieved, and it maintains that commitment until it believes either that the goal is achieved or that it is unachievable. That second exit is the terminal state every current agent framework lacks. It was written down thirty-six years ago in the field's flagship journal.
- · 7 min read
Two Correct Termination Proofs Compose Into a Hang
Your planner stops. Your executor stops. Both proofs are valid and neither is wrong. Put them in a loop together and the system runs forever, because termination is not a property that composes. Podelski and Rybalchenko formalised the fix in 2004, and it explains exactly why per-component monitoring cannot see a hang that lives between components.
- · 8 min read
The Verifier That Verifies Nothing
Naming a pipeline stage 'verifier' does not make it one. If the stage is another model call asked whether the output looks right, it shares the generator's blind spots, produces an opinion rather than a check, and has almost certainly never rejected anything. There is a ladder of verifier strength with five rungs, and most production systems are on the bottom one while believing they are near the top.
- · 8 min read
The Third Option: Large Integrator, Narrow Specialist, or One Accountable Owner
An organisation with a hard cross-layer production problem weighs four options, and each is structurally wrong for it in a different way. The large integrator is broad and shallow with hand-offs at every seam. The narrow specialist is deep in one layer and hands off at its edge. The AI vendor sells the five percent. Staffing gives you hands without an owner. This is what the fifth option is and when it is not the answer.
- · 7 min read
The Delete-the-Variable Test
A ten-second check that tells you whether a variable is doing real work or is scaffolding: delete it, rerun, and see what changes. Delete TCP's congestion window and the internet falls over, which it did in 1986. Delete your agent's iteration cap and the trace does not flinch. The test has one important limit, and knowing it is what stops the test becoming a licence to remove your own safety nets.
- · 8 min read
Rebuild or Repair? How to Tell Before You Approve Six Months
Two contractors quoted four and six months to rebuild a model. The actual fix took five days and was not the model. The decision between rebuilding and repairing is not a judgement call, it is a diagnosis you can run in an afternoon, and running it first is the difference between a five-day fix and six figures spent reproducing the same failure.
- · 10 min read
I Read the Stop Condition in Nine Agent Frameworks. The Third Column Is Empty.
Nine agent frameworks, from nine organisations, in different languages, with no shared lineage. Every one decides whether to keep going using the same two ingredients: the model's opinion that it is not finished, and a counter. Not one of them checks a measure of the remaining work. The third column of the audit is empty, and it is empty for a reason that is not laziness.
- · 7 min read
The Five Ways a Loop Fails to Stop
Floyd gave us the conditions a terminating loop has to satisfy in 1967. Negate them one at a time and you get an exhaustive list of the ways termination can fail, which turns a vague symptom into five named diagnoses with five different fixes. Your agent is almost certainly number four, and number four is the only one that looks like success.
- · 7 min read
Bounded the Wrong Thing: What TCP's 1988 Fix Teaches Your Agent Loop
TCP already had a bound before congestion collapse. The receive window was real, correctly enforced, and nobody had a bug in it. The internet fell over anyway, because that window protected the receiver and the thing failing was the network in between. Jacobson did not delete it. He added a second window measuring the thing that was actually going wrong and took the minimum of the two.
- · 7 min read
Your Agent Is Missing a State: Done, Gave Up, Killed
Chess has a draw, which is not a loss. A solver returns unknown, which means it ran out of budget and not that no answer exists. A database names a deadlock victim so the transaction knows it was killed. Your agent has two states, done and killed, and when the cap trips the run must be filed as one of the two it has a name for, so it gets filed as done. Everything downstream believes it.
- · 9 min read
You Cannot Hire a Gap: Why Cross-Layer Failures Have No Owner
The analog engineer designed a correct reference loop. The actuation engineer designed a correct current source with a safety limit. The mechanical engineer placed the electrodes correctly. Every one of them did excellent work inside their lane, and the device failed, because the failure lived in the loop connecting all three and that loop was on nobody's schematic. A seam is not a gap in anyone's competence. It is the region between all of your competent people.
- · 6 min read
Wrong vs Load-Bearing Wrong: When One AI Answer Becomes a Foundation
A system that is wrong costs you one answer. A system that is load-bearing and wrong costs you everything standing on that answer. In a trusted internal tool the output is not the end of anything, it is the start of a chain: a wrong answer becomes a wrong action, the action becomes a commitment, and no stage downstream ever rereads the source. Nothing on your dashboard tells you which of the two you have.
- · 7 min read
A Time Fuse and an Iteration Fuse Are Not the Same Thing
A hung tool call freezes the iteration counter forever, so a max-iteration cap can never fire on a hang. Only a clock catches it, which is exactly why every embedded watchdog counts time rather than iterations and runs on its own oscillator. Too slow is a hang and too fast is a spin: two opposite edges, two different instruments, and most teams have shipped two copies of the same half.
- · 10 min read
The Model Is Five Percent: Where Production AI Actually Fails
Across a decade of systems in defense, medical devices, industrial automation, fintech, and infrastructure at fleet scale, the failure is almost never in the model. It is in the ninety-five percent underneath: the pipelines, the protocols, the drivers, the firmware, the silicon, and above all the seams between them. The clearest proof is that the identical failure shows up in systems with no model in them at all.
- · 7 min read
The Cloud Hides Your AI's Real Failures
Four properties of every model are always present and almost never visible: it extrapolates past its data, it discards most of your input, it is memory-bound on a shared bus, and its confidence off-distribution is undefined. A cloud environment hides all four, because a retry, a redeploy, a bigger box, and a human on a dashboard tomorrow each absorb one. Take the second chance away and they stop hiding at once.
- · 8 min read
How to Build a Test Set That Actually Predicts Production
A test set drawn at random from your training data measures how much your test set resembles your training set, not whether the system works. Building one that predicts production means holding out whole groups rather than random samples, choosing the grouping variable that matches how your system will actually be deployed, sizing the held-out groups honestly, and checking for the leakage that inflates a score while nothing was learned.
- · 7 min read
Shadow and Canary: Shipping an AI Change Without Betting the Business
Two unglamorous techniques between a change and everyone. Shadow runs the new version against real traffic without showing anyone the results, so you find out how it behaves on production inputs before a single user sees it. Canary releases it to a small fraction with an automatic undo. The alternative, which is what most teams do, is to flip it on for everyone and hope.
- · 8 min read
Recall Is the Silent Killer of Your RAG System
If the answer lives in a document your system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened, so it answers confidently from whatever it did find, and the output looks exactly like a good answer. Recall is the one failure that leaves no trace in the output, which is why it is the first thing to measure and the last thing anyone does.
- · 7 min read
How One Bad Input Poisons an Entire Agent Chain (and the Fix)
One step missed a single document. That assumption reached the planner, which generated three subtasks that each inherited it, and their outputs became the next step's input. Five steps down, the output had nothing to do with reality, confident and completely wrong, with every individual step reasoning flawlessly. The fix was not the model: letting each step reject a bad input instead of swallowing it cut the cascade by 82 percent.
- · 7 min read
You Can't Fix What You Can't See: Instrumenting an AI Agent
A real system had 11 internal steps and logged exactly one thing: did the whole run work, yes or no. So when it failed it looked random, and good engineers said it fails randomly and we cannot reproduce it. There is no such thing as random failure, only failure you cannot see. Instrumenting every step took two days and showed 85 percent of failures coming from one step, which then took two days to fix.
- · 8 min read
Every Stopping Rule Needs a Model of the Task: Eighty Years of Evidence
Munitions inspection in 1945, early stopping in clinical trials, Bayesian optimization, multi-armed bandits, index policies, and proof assistants that will not compile until you supply the measure yourself. Seven fields over eighty years that do not read each other's papers, and every one needed a number describing the work rather than the effort. Your LLM agent has a model of language and no model of the task, which is exactly why its stopping rule is missing.
- · 7 min read
Every Metric Is a Proxy: Goodhart's Law for AI Systems
You cannot read forty thousand answers, so you appoint one number as a messenger and send it to check the truth. Every dashboard number is that: a stand-in for something you actually care about but cannot watch directly. The gap between the number and the thing is where trouble hides, and the moment the number becomes the goal, people find ways to move it that have nothing to do with what you wanted.
- · 7 min read
Durable Execution Is Not Exactly-Once (and What to Do Instead)
Durable execution replays a workflow and skips the steps that already completed, which narrows the window in which a side effect can run twice. It does not close it, because the gap between performing an effect and recording that you performed it cannot be made atomic across a network boundary. Nothing provides exactly-once. What you get is at-least-once plus a way to make the second one harmless.
- · 7 min read
Agents Take Actions That Can't Be Undone: Designing for Irreversibility
A chatbot writes words, and if it is wrong you delete it. An agent acts: it sends the email, charges the card, deletes the record, places the order, and those actions turn one way at machine speed. In 2012 an automated trading system shipped a bad deployment with no kill switch and lost 440 million dollars in about five minutes. The code was the last domino. The failure was the system that let it run with no brakes.
- · 7 min read
Your Bill of Materials Passes and Your Board Is Still Wrong: Topology vs Parts
Two current-limit resistors in series for redundancy is good design. The same two resistors, both the correct value, placed in parallel instead, quarters your limiting and roughly doubles the current. Every part inspection passes, because both resistors are the right value and the bill of materials is correct. The failure is not in the parts, it is in the topology, and your incoming checks do not look at topology.
- · 8 min read
Before You Blame the Model: The Four Boring Checks
When an AI system works in testing and fails in production, the model is the last thing to suspect, not the first. Four checks rule out the plumbing before you touch anything interesting: is the same model actually running, is it getting the same input, is the preprocessing identical, and is something quietly timing out or falling back. Most reported model failures die at one of these four.
- · 7 min read
Why You Can't Let an AI Grade Its Own Homework
Use the same model to write an answer and to judge it, and the loop closes on itself: it keeps the same blind spots twice, and it goes easy on itself in ways that are measurable rather than figurative. It prefers longer answers and its own writing style, and scores those higher. The single sharpest question to ask anyone building your evaluation is what is doing the checking.
- · 13 min read
Why Your AI Agents Fail, and How to Fix It
AI agents almost never fail because of the model. Chain enough steps and the math turns against you: at 95% reliability per step, a 20-step agent finishes under 40% of the time, and the failure lives in the coordination between steps, not inside any one of them. The fix is system engineering, verification, state, and guardrails, not a better model.
- · 13 min read
Why Your AI Agent Won't Stop (and Why Capping the Loop Doesn't Fix It)
An AI agent can run green for 264 hours and bill 47,000 dollars with every single call bounded, because the iteration cap is a fuel gauge, not a progress bar. It measures how much you have spent, not how much work is left, so 'I gave up' and 'I am done' come back identical. Stopping reliably requires a number that measures the task itself, computed without asking the model.
- · 12 min read
Why Your AI Agent Double-Executes: Retry Is a Second Execution, Not a Second Chance
When an agent charges a card twice or sends an email twice, nothing is broken to go find: no error, no red in the logs, the action was correct and simply ran twice. Duplicates enter through four doors, retry, redelivery, redispatch, and double trigger, but a duplicate only hurts you when there is no gate in front of the side effect. The fix is one idempotency key, identical no matter which door it came through.
- · 12 min read
Why Your AI Agent Reports Success But Nothing Ran: Five Ways a Tool Call Dies
When an agent says it called a tool, the run comes back green, and nothing happened, the model almost never lied. A tool call is just text the model hands off, and it gets dropped somewhere in the plumbing between the model and the tool, in one of five places: streaming, index collision, proxy translation, the wrong finish flag, or no check at the end. A bigger model fixes none of them.
- · 11 min read
The Eighty Percent That Separates an AI Agent Demo From a Platform
Building the agents is about 20% of the work. The 80% that decides whether an agent survives production is the part everyone skips: reconciliation when systems disagree, confidence thresholds that return no answer instead of a wrong one, validation before any real action, and the cost and observability work. A better model does not touch any of it, and all of it is buildable today.
- · 10 min read
Retrieval or Generation? Why You Can't Tell Which Half of Your RAG System Broke
A wrong RAG answer is not one failure, it is two: the search pulled the wrong pages, or the search worked and the model ignored them. From the output they look identical, same confident tone, same citation, but they need opposite fixes. And one blended quality score averages the two into a green number that hides which half is on fire.
- · 9 min read
Index Freshness and Drift: Why a Working RAG System Quietly Goes Wrong
A RAG system that works today degrades quietly as its documents change. There are two kinds of drift: the easy one, where the questions start looking different, and the nasty one, where the questions look identical but the right answer changed underneath. The fix is re-checking on a schedule against a baseline, and never auto-rebuilding when the alarm fires, because half the time it is a data-plumbing break, not a real change.
- · 15 min read
How to Fix a Broken RAG System (Without Rebuilding the Model)
A RAG system that scored 95% in testing and collapses in production almost never needs a model rebuild. The failure is usually in one corner of the pipeline, and the fix is four cheap steps: slice the score to find the broken corner, measure recall precision and faithfulness separately, build a check that is not the thing being checked, and re-check on a schedule.
- · 12 min read
Confidence Is Not Evidence: Why an AI Model Can't Tell You When to Trust It
A model is geometry fit to data, so past the edge of its training it does not stop, it extrapolates, applying a flat rule that was anchored to nothing, with full confidence. Its certainty is not evidence, because standard training optimizes knowing the answer and never optimizes knowing whether it knows. The decision to trust it has to live outside the model.