Tagged systems
28 pieces
- · 6 min read
Durable Execution, Idempotency, or Both
A workflow engine guarantees your process resumes. It does not guarantee your side effect happened once. Those are different properties solving different halves of the same failure, and buying the first while assuming it delivers the second is the most common way a durable system still double-charges someone.
- · 6 min read
Perception Accuracy Is Not Control Safety
A 99 percent accurate detector is not a 99 percent safe robot, and the gap is not a matter of degree. Safety depends on which errors occur, how they are distributed in space and time, what the system does when it is uncertain, and whether the safe behaviour is enforced somewhere the perception stack cannot override.
- · 7 min read
Perception to Control: The Handoff Nobody Owns
The perception team ships a detector that meets its accuracy target. The controls team builds a controller that is stable given its inputs. Both are right and the robot behaves badly, because what crosses between them is not a detection, it is a detection with a latency, a coordinate frame, a timestamp, and an uncertainty, and only one of those five is usually specified.
- · 7 min read
Reconciliation Is Not a Nice-to-Have
Prevention mechanisms fail silently. Reconciliation is the only part of a money-moving system that can tell you they did, and it is the component most often deferred, because it produces no feature and its value is invisible on every day it finds nothing.
- · 7 min read
Idempotency for Money Movement
An idempotency key is not a header you add. It is a claim about what constitutes the same operation, and almost every double-charge traces to that claim being wrong: the key was generated per attempt instead of per intent, or the deduplication window closed before the retry arrived, or the key covered the request and not the effect.
- · 7 min read
Ship It Once, Reach It Never: Designing for Hardware You Cannot Touch
When a unit leaves the building, the set of decisions you can still change shrinks to almost nothing. The design question becomes which properties must be right at manufacture and which can be deferred, and getting that split wrong is the most expensive mistake in connected hardware because it is discovered across an entire production run at once.
- · 7 min read
Systems You Cannot Take Offline to Fix
Most engineering practice assumes you can stop the thing to work on it. Remove that and the entire method changes: you diagnose from passive observation, you change one variable per window, you validate in shadow before you commit, and you design every modification to be abandonable halfway through.
- · 7 min read
Desired State: How Fleet Systems Made Double Execution Impossible
Infrastructure engineering solved a problem AI agents are currently rediscovering. Declaring the state you want, rather than the action you want taken, makes repeating an operation harmless by construction. The interesting part is not the idea. It is exactly which properties do the work, because two of the three are usually missing when people borrow the pattern.
- · 7 min read
Fleet Scale Changes the Failure Mode, Not Just the Number
A platform that works at a hundred nodes and fails at twenty-five thousand did not run out of capacity. It met failures that do not exist at small scale: monitoring that damages what it monitors, silence that no poller notices, and heterogeneity that only appears once the sample is large enough to contain it.
- · 7 min read
PLC to Model: Where Industrial AI Actually Breaks
The model is rarely the problem on a factory floor. The problem is the path between the controller and the model: a protocol implemented as the vendor felt like implementing it, a timing budget with the round-trip inside it, a controller that cannot be taken out of service, and a boundary that belongs to neither the automation team nor the data team.
- · 7 min read
Signs the Failure Is in the Seams, Not the Model
Every component passes its own tests. Every team's investigation ends politely at their boundary. The system does not work. Five signs tell you the failure lives between components rather than inside one, and they matter because a seam failure bought as a component engagement is money spent on the wrong question.
- · 7 min read
The Five Ways a Loop Fails to Stop
Floyd gave us the conditions a terminating loop has to satisfy in 1967. Negate them one at a time and you get an exhaustive list of the ways termination can fail, which turns a vague symptom into five named diagnoses with five different fixes. Your agent is almost certainly number four, and number four is the only one that looks like success.
- · 7 min read
Two Correct Termination Proofs Compose Into a Hang
Your planner stops. Your executor stops. Both proofs are valid and neither is wrong. Put them in a loop together and the system runs forever, because termination is not a property that composes. Podelski and Rybalchenko formalised the fix in 2004, and it explains exactly why per-component monitoring cannot see a hang that lives between components.
- · 7 min read
Bounded the Wrong Thing: What TCP's 1988 Fix Teaches Your Agent Loop
TCP already had a bound before congestion collapse. The receive window was real, correctly enforced, and nobody had a bug in it. The internet fell over anyway, because that window protected the receiver and the thing failing was the network in between. Jacobson did not delete it. He added a second window measuring the thing that was actually going wrong and took the minimum of the two.
- · 7 min read
The Delete-the-Variable Test
A ten-second check that tells you whether a variable is doing real work or is scaffolding: delete it, rerun, and see what changes. Delete TCP's congestion window and the internet falls over, which it did in 1986. Delete your agent's iteration cap and the trace does not flinch. The test has one important limit, and knowing it is what stops the test becoming a licence to remove your own safety nets.
- · 8 min read
The Third Option: Large Integrator, Narrow Specialist, or One Accountable Owner
An organisation with a hard cross-layer production problem weighs four options, and each is structurally wrong for it in a different way. The large integrator is broad and shallow with hand-offs at every seam. The narrow specialist is deep in one layer and hands off at its edge. The AI vendor sells the five percent. Staffing gives you hands without an owner. This is what the fifth option is and when it is not the answer.
- · 7 min read
Your Agent Framework Was Specified in 1990 (And Is Missing the Same State)
In 1990 Cohen and Levesque formally specified what it means for an agent to hold a goal: it believes the goal is not yet achieved, it wants it achieved, and it maintains that commitment until it believes either that the goal is achieved or that it is unachievable. That second exit is the terminal state every current agent framework lacks. It was written down thirty-six years ago in the field's flagship journal.
- · 10 min read
I Read the Stop Condition in Nine Agent Frameworks. The Third Column Is Empty.
Nine agent frameworks, from nine organisations, in different languages, with no shared lineage. Every one decides whether to keep going using the same two ingredients: the model's opinion that it is not finished, and a counter. Not one of them checks a measure of the remaining work. The third column of the audit is empty, and it is empty for a reason that is not laziness.
- · 9 min read
You Cannot Hire a Gap: Why Cross-Layer Failures Have No Owner
The analog engineer designed a correct reference loop. The actuation engineer designed a correct current source with a safety limit. The mechanical engineer placed the electrodes correctly. Every one of them did excellent work inside their lane, and the device failed, because the failure lived in the loop connecting all three and that loop was on nobody's schematic. A seam is not a gap in anyone's competence. It is the region between all of your competent people.
- · 10 min read
The Model Is Five Percent: Where Production AI Actually Fails
Across a decade of systems in defense, medical devices, industrial automation, fintech, and infrastructure at fleet scale, the failure is almost never in the model. It is in the ninety-five percent underneath: the pipelines, the protocols, the drivers, the firmware, the silicon, and above all the seams between them. The clearest proof is that the identical failure shows up in systems with no model in them at all.
- · 8 min read
Every Stopping Rule Needs a Model of the Task: Eighty Years of Evidence
Munitions inspection in 1945, early stopping in clinical trials, Bayesian optimization, multi-armed bandits, index policies, and proof assistants that will not compile until you supply the measure yourself. Seven fields over eighty years that do not read each other's papers, and every one needed a number describing the work rather than the effort. Your LLM agent has a model of language and no model of the task, which is exactly why its stopping rule is missing.
- · 7 min read
Your Agent Is Missing a State: Done, Gave Up, Killed
Chess has a draw, which is not a loss. A solver returns unknown, which means it ran out of budget and not that no answer exists. A database names a deadlock victim so the transaction knows it was killed. Your agent has two states, done and killed, and when the cap trips the run must be filed as one of the two it has a name for, so it gets filed as done. Everything downstream believes it.
- · 7 min read
You Can't Fix What You Can't See: Instrumenting an AI Agent
A real system had 11 internal steps and logged exactly one thing: did the whole run work, yes or no. So when it failed it looked random, and good engineers said it fails randomly and we cannot reproduce it. There is no such thing as random failure, only failure you cannot see. Instrumenting every step took two days and showed 85 percent of failures coming from one step, which then took two days to fix.
- · 7 min read
Agents Take Actions That Can't Be Undone: Designing for Irreversibility
A chatbot writes words, and if it is wrong you delete it. An agent acts: it sends the email, charges the card, deletes the record, places the order, and those actions turn one way at machine speed. In 2012 an automated trading system shipped a bad deployment with no kill switch and lost 440 million dollars in about five minutes. The code was the last domino. The failure was the system that let it run with no brakes.
- · 7 min read
How One Bad Input Poisons an Entire Agent Chain (and the Fix)
One step missed a single document. That assumption reached the planner, which generated three subtasks that each inherited it, and their outputs became the next step's input. Five steps down, the output had nothing to do with reality, confident and completely wrong, with every individual step reasoning flawlessly. The fix was not the model: letting each step reject a bad input instead of swallowing it cut the cascade by 82 percent.
- · 6 min read
Wrong vs Load-Bearing Wrong: When One AI Answer Becomes a Foundation
A system that is wrong costs you one answer. A system that is load-bearing and wrong costs you everything standing on that answer. In a trusted internal tool the output is not the end of anything, it is the start of a chain: a wrong answer becomes a wrong action, the action becomes a commitment, and no stage downstream ever rereads the source. Nothing on your dashboard tells you which of the two you have.
- · 13 min read
Why Your AI Agent Won't Stop (and Why Capping the Loop Doesn't Fix It)
An AI agent can run green for 264 hours and bill 47,000 dollars with every single call bounded, because the iteration cap is a fuel gauge, not a progress bar. It measures how much you have spent, not how much work is left, so 'I gave up' and 'I am done' come back identical. Stopping reliably requires a number that measures the task itself, computed without asking the model.
- · 13 min read
Why Your AI Agents Fail, and How to Fix It
AI agents almost never fail because of the model. Chain enough steps and the math turns against you: at 95% reliability per step, a 20-step agent finishes under 40% of the time, and the failure lives in the coordination between steps, not inside any one of them. The fix is system engineering, verification, state, and guardrails, not a better model.