Weekly long-form on why production systems fail and what it takes to make them hold, model to metal. Written by the person who does the work, not by a marketing team summarizing it.
An entitlement system written up as thirteen modules of screens has one real deliverable, and it is an arithmetic identity: allocated, less used, less pending, equals avail...
Integrating providers inline means every provider's schema, auth, rate limits and failure modes spread through your application, and one provider's outage empties the page...
A three-layer record holds an append-only source layer, a governed derived layer, and a revisable current-state layer with history. The rule between them is that the derive...
A headless agent run that hits a rate limit can exit with a success status and a zero exit code, having written nothing at all. Every reliability feature that reads the age...
When a counterparty can accept an executable quote from a phone, the accept action will be repeated. A tap on poor signal that appears to do nothing gets tapped again, clie...
When an assistant answers questions about live records, the engineering is the permission boundary, not the retrieval. In a Postgres demonstrator with a correctly written r...
When a product is slow under load and adding servers makes the economics worse without fixing the latency, the bottleneck is the work inside one request rather than the cap...
A demonstration shows an agent pipeline working on the path its author chose. A drill breaks it on purpose and records what the system does. Six drills cover most of what g...
When settlement and reporting each maintain their own copy of what happened, they will eventually disagree and nobody can say which one is right. Derived as projections of...
Giving an agent a read-only database role stops it writing. It does nothing about what it can see, because Postgres owner roles and the pg_read_all_data role bypass row-lev...
A reference set assembled from a pipeline's own extracted output scores that pipeline against itself. Wherever the extraction is wrong, the reference is wrong in the same d...
The widely copied hybrid search recipe pairs a vector index with Postgres full-text search and fuses the two with reciprocal rank fusion. Run a Chinese query against it and...
In at least one widely used workflow engine, cancellation is delivered only to work that heartbeats. Wrap a command-line process that never reports in, and the cancel contr...
A Postgres view executes with its definer's privileges unless security_invoker is set, and that setting is off by default. Every view created before somebody thought about...
A harness that has never detected a known fault has not been shown to detect anything. Injecting deliberate, known faults is the step that converts a green suite into a sui...
A platform that serves public market information and private executable quotes from one state core has to check its boundary against the object being requested, not against...
A balance held in a column is a cache with no source behind it. Every correction becomes an edit, and nothing in the system can reconstruct how the number reached its curre...
When a cheap option carries a risk the user cannot see, ranking it first and warning in the fine print puts the trap exactly where trust is highest. Treat every option as a...
The site, the payment processor and your records each work correctly on their own, and the money that goes missing is lost in the gaps between them. Confirming on the landi...
Add a WHERE clause to a vector search and the index does not search harder to compensate. It walks its usual neighbourhood, most of what it finds fails your filter, and you...
For a transactional product, the shortest reliable path to one real settled transaction is worth more than a broad surface built before anything clears. Breadth first means...
A tool that snapshots its own view of a workspace cannot capture changes made outside that view, such as a file moved or generated by a shell command. Restoring the checkpo...
Major generative AI providers' terms bar applications that are directed at, or likely to be used by, people under eighteen. The moment a product defines a Student or a Chil...
Published benchmarks of agents on realistic computer-use tasks report completion rates in the low tens of per cent, while humans on the same tasks sit near ceiling. That ga...
A platform that lets a counterparty accept a binding quote without a human looking at it has not removed the gate, it has moved it into code. Every check the trader used to...
Event sourcing assumes you can replay the source and get the same derived state. A model call in the derivation breaks that assumption, because the lowest temperature setti...
A multi-stage pipeline judged only on its combined output cannot tell you which stage contributed nothing. An extraction error, a dead retrieval branch, a fusion step absor...
A strict JSON schema on an extraction pipeline validates type, range and required fields. It cannot see that the number 12 in a table headed in ten-thousands means 120,000....
When a platform transacts and a separate system holds the customer relationship, retro-fitting the sync between them costs far more than building it with the first transact...
When a person certifies a figure your system produced, extraction accuracy is the wrong metric. The document your pipeline reads can be perfectly extracted and still contai...
Shifting an allocation from one commitment to another moves real exposure, so it has to be recorded as its own transition with its own entry. Implemented as an edit to two...
For most model-backed features the output side dominates the cost, because output tokens are priced at a multiple of input tokens and because caching makes repeated input c...
Calibration in aggregate and the ability to rank individual items are two different properties, and a score can have the first without the second. In a lab reproduction, a...
Switching a calculation engine to decimal arithmetic fixes nothing if the numbers arrive as floats. A spreadsheet stores numeric cells as IEEE-754 doubles, the usual datafr...
When a team buys rapid prototyping, the thing they actually need is a decision: build this or do not. The cost of a wrong keep, carrying a bad idea into a full build, excee...
When several integrated systems each hold a version of the same fact and each can write it, there is no answer to which one is right. Not a hard answer, no answer: there is...
A detector that failed to fire has at least three causes that are indistinguishable from outside the system. The signal was removed by a transform before it arrived, the th...
An administrative correction applied by editing a figure destroys the ability to reconstruct how that figure reached its value, and it hides the fact that a correction happ...
Published work on audio watermarking reports detection performance near 0.97 on clean audio falling toward chance, about 0.57, under certain neural codecs. That is the tran...
Kill a worker mid-job and the two most common database-backed job queues behave in opposite ways. One re-runs the job after a fixed interval whether the worker is alive or...
A number shown to somebody who will be held accountable for it has to be fresh, entitled and real. Fresh needs an alarm that fires on absence. Entitled needs a check at the...
Under ordinary database isolation levels two concurrent requests can both read the same available balance, both find it sufficient, and both commit, leaving the balance neg...
On a one-week prototype the schedule risk is almost never the code. It is platform gatekeeping: code signing and notarisation, a mobile SDK version mismatch inside the clie...
A feed that stops sends no out-of-range value, so a threshold alarm has nothing to fire on and the last good number keeps being served as though it were current. The only t...
Request-response interface description formats cannot express an event-driven contract. There is no caller, there is no meaningful response, delivery is at-least-once, and...
A model that predicts close to the population mean can place most of its points in a clinical error grid's acceptable zones and post a respectable error percentage, while e...
Hosted model providers retire models on published schedules, and some deployment types upgrade automatically. For software whose behaviour has been validated, a model chang...
Photograph the same sample five times without touching anything and the derived value will differ across the five images. If your decision threshold is narrower than that s...
A camera's response to light is deliberately nonlinear, so the pixel value in a JPEG is not proportional to the light that produced it. Any intensity measurement taken befo...
Cut a sensor's supply rail and it can stay awake, drawing current through the pull-up resistors on a shared bus that never switches off. The current arrives through the par...
When a product runs many campaigns or tenants, copying the working flow per campaign is the default, and the copies drift. A reliability fix lands in whichever copy somebod...
Replacing a long-lived platform gets scoped as a greenfield build because the new stack is the exciting part. What actually has to move is fifteen years of business rules e...
Carry-over, expiry and lapse at a period boundary are derivation rules applied at a date. Implemented as a destructive reset that overwrites the previous period's state, th...
In a layered product, risk is not where the engineering is hardest. It is where there is neither a worked reference implementation nor a mechanism that fails loudly. In a c...
Data residency for model inference is frequently not selectable in the way teams assume. Cross-region routing can place inference outside an intended boundary, and some glo...
A product that must grow from a seed either lays its reliability core at phase one, or rebuilds at every growth step and re-litigates the same reliability questions four ti...
Optical heart-rate algorithms separate the cardiac signal from motion by picking the dominant periodic component. When step frequency converges on heart rate, which happens...
A spend limit on an automated run is a control only if it accumulates across the whole task. On at least one widely used agent runner the cap applies per invocation, so res...
In long-lived business platforms a large share of policy sits in stored procedures, triggers, constraints, defaults and view definitions rather than in application code. A...
On one major mobile platform, a web app installed to the home screen has storage isolated from the browser. The cookie jar, local storage and database are a separate contai...
A calendar convention read as the wrong system shifts every date by a fixed offset that can be centuries, and nothing rejects the result because the output is a valid date....
Audio feature extraction has at least six parameters whose defaults differ between libraries, including the mel scale convention itself. A mismatch between the path used in...
On a tightly coupled memory architecture, two real-time paths that share a memory bank can collide, and the symptom you see is a buffer underrun. You can chase that symptom...
When quote, acceptance and settlement exist as three screens calling services ad hoc, the acceptance and the trade record can disagree, and a double tap can execute twice....
When one user request has to become many calls to a paid provider, cost and latency scale with the number of candidates rather than with the number of requests. The fix is...
A row-level security policy that lets a user update their own row is correct for a display name and catastrophic for a plan tier or a role. The policy decides which rows a...
The cheapest risk reduction available in a migration is a first gate that runs entirely against a database copy and ships no features. Because there is no operational risk,...
The governing international standard for medical device risk management requires each risk control to be verified twice: once that it was implemented, and once that it is e...
When final acceptance happens on hardware you will never physically touch, that stage is the one you control least. On a fixed-price engagement, weighting milestones by eff...
An indicative value and a committed value are different objects with different tolerances, and the cheapest way to break a product is to let one become the other. A cached...
Running independent calls concurrently is the right instinct, and unbounded concurrency converts a slow path into a failing one. Without a ceiling, a wide request exhausts...
A convincing mechanical heartbeat is asymmetric and punctuated: a fast contraction over roughly one third of the cycle, a brief hold, a slower release, then a real pause ta...
A product that offers a clever alternative to an obvious option has to show it beats that option, on the same inputs, at the same moment. Ranking your own candidates agains...
An operator who buys and sells on their own book is not an intermediary moving other people's goods. What they hold is exposure, and exposure is not a quantity. A warehouse...
A business rule that only fires on a condition absent from your copied data will not be discovered by reading the schema, however carefully you read it. That is the structu...
When a specification carries a large blocking acceptance-test suite and a global safety gate, scoping the first release by feature value means every later release renegotia...
One threshold applied everywhere is simultaneously too strict where conditions are easy and too loose where they are hard, so it rejects good options and admits dangerous o...
Detaching or stopping the control signal to a servo does not remove power from it. The internal controller keeps running off the supply rail, and under mechanical load the...
An integration layer that improved API success, end-to-end failure, duplicate results and empty responses did so through four separate mechanisms. Reported as one number, y...
Two records from different sources cannot be compared until they share a shape, so deduplication is downstream of normalisation whether or not anyone planned it that way. T...
Inbound webhooks from major messaging and social platforms retry over an extended window and typically offer no way to request message history afterwards. That pairing prod...
Current consumption in a well-designed low-power device spans several orders of magnitude across a duty cycle, from microamps asleep to tens of milliamps transmitting. A be...
Every product has one thing certain to grow: the list of cities, currencies, carriers, document types or tenants it supports. Where that list lives is decided on day one an...
In a regulated product, which system holds the official copy of each piece of data sets the compliance posture, the monthly cost, and whether later integrations are additio...
Sales, labour and cost of goods each mean something specific in food and beverage, and a generalist gets all three wrong in directions that look plausible. Sales taken gros...
For products whose value is access to someone else's data, the risk that ends the project is not the engineering. It is whether you can obtain the data at all, on terms tha...
Incremental replacement behind an existing system means two implementations coexist for the whole migration, with a routing seam between them. That is a real cost rather th...
On at least one major connected-device platform, a firmware update progress figure reaching 90 percent means the download completed, not that the install completed. Teams r...
A non-technical client cannot evaluate progress by looking at the work, so milestones built around internal stages leave them trusting rather than knowing. Structure the st...
Three systems, each owned, each correct by its own contract, each reporting success. The failure lives in the relationship between two of them, and relationships are not as...
The same piece of work can be bought from a platform, an agency, a specialist, a generalist or in-house, and the engineering is similar in all five. What differs is who fin...
Platforms that enforce a strictly-higher-version rule make conventional rollback impossible, because the device will refuse any image numbered below what it is running. The...
No mobile browser on one major platform exposes the interface a web page needs to reach a Bluetooth peripheral. If a connected device is anywhere on the roadmap, the produc...
Deduplicating on an event identifier and deriving state from an append-only ledger gives you exactly-once behaviour over an at-least-once unordered stream. That is the stro...
When software controls a physical queue of people, a failure has consequences that are not data consequences. Offline-first operation at the counter, where each terminal ke...
An operator reading a figure cannot tell a reconciled number from an unreconciled one, so a plausible narrative over numbers that do not tie back is indistinguishable from...
Attribution reconstructed after the fact from session data, timing or traffic patterns is a model presented as a measurement, and it is confidently wrong in the cases that...
A rollout ladder controls how many devices a build reaches, and that is the entire extent of what it does. It cannot make the build safe. A one percent rollout of firmware...
Owning one layer of a telemetry chain means shipping a dashboard that is confidently wrong in ways nobody upstream can see. Four proven hops and one unproven does not give...
A specification document carries forensic information its authors did not intend to send. The rendering engine and version reveal how it was produced. Route names in a modu...
When nobody knows whether a critical fact is true, shape the engagement so that both answers lead to a working product. One monitoring project did not know whether a contro...
When firmware, gateway, cloud and dashboard are split across contractors, the failures land in the seams between them, where each side can demonstrate that its own part wor...
Class A is no possible injury, Class B non-serious injury, Class C death or serious injury. Verification obligations rise with the class: more rigorous unit, integration an...
Branch coverage is highly recommended at ASIL B, C and D; MC/DC and fault injection become highly recommended at ASIL D. The hardware side permits only about 1 percent resi...
Branch coverage on D = A and B is satisfied by two tests, and a bug that ignores B passes both. MC/DC requires each condition to independently flip the outcome, needing thr...
A reused module carries a correctness argument scoped to its old environment. Ariane 5 lost a launcher to a conversion that was safe within Ariane 4's velocity range, and t...
A coverage guided fuzzer mutates inputs to drive new paths, and a sanitizer catches the out of bounds read that a functional test never notices. Neither half is sufficient...
Instead of hand picking examples, state a property such as declared length never exceeds actual length, and let the framework generate hundreds of inputs and shrink any fai...
Corrupt a sensor value, force an error return, flip a bit, drop a packet, delay an interrupt, and confirm the system detects it. This is the MCAS lesson, ISO 26262 makes it...
Naive full repo mutation runs the suite once per mutant and takes hours. Google's method makes it tractable at roughly two billion lines: mutate only changed covered lines,...
Delete one line of production code, or flip one condition, and run your suite. If nothing goes red, that part of your suite is a decoration. It is a manual one mutant mutat...
Before trusting any green check, make it go red on a known broken input. Three concrete forms: inject a known defect and confirm the suite fails, keep a must-fail case in C...
Four industrial teams saw pre release defect density fall between 40 and 90 percent, at a subjective cost of 15 to 35 percent more initial development time. A controlled st...
A mutation that produces behaviourally identical code can never be killed, so it inflates the surviving count and makes 100 percent unreachable. Prevalence runs 4 to 39 per...
Across 357 real faults and 230,000 mutants, a coupling relationship held for 73 percent of real faults, and mutant detection tracked real fault detection more strongly than...
A test can execute a line and assert nothing about it. That line counts as covered and is completely unprotected, and no coverage tool can tell the difference. Assertion de...
About 16 percent of Google's 4.2 million tests are flaky and 84 percent of pass to fail transitions are not real regressions. Only 1.23 percent of tests have ever found a b...
Control for suite size and the correlation between coverage and fault detection drops to low or moderate. Against real post release defects it is statistically insignifican...
To a diligence engineer, a suite nobody has watched fail is unproven, and unproven is indistinguishable from absent. Seven red flags reprice deals and each maps to a public...
The Therac-25 removed the hardware interlocks its reused code assumed and gave the job to software. A race condition reachable by fast operator edits, plus a one byte count...
MCAS depended on one angle of attack sensor for a function Boeing's own assessment classified as hazardous, a classification that normally precludes single sensor dependenc...
OpenSSL's heartbeat handler copied as many bytes as the client declared without checking that against what the client actually sent, returning up to 64 KB of adjacent memor...
A ground software module computed thruster impulse in pound force seconds while the interface specification required newton seconds. Navigation consumed it as newton second...
A technician copied new code to eight servers by hand and one did not receive it. A repurposed feature flag then reactivated dormant routing logic on that one server. Rough...
The inertial reference module was reused unchanged from Ariane 4 and had a flawless demonstration record. Ariane 5 produced horizontal velocity values roughly five times la...
The template defined twenty one input fields and the interpreter supplied twenty. Earlier instances wildcarded the twenty first, so the mismatch could not manifest and four...
When a system cannot proceed safely, it has two options: hand the decision to a person, or take a defined safe action itself. Which one is correct is decided by whether a p...
A passing suite, a clean demo and green CI are evidence of nothing until somebody has watched the check go red on a known broken input. Passing is a property of the test, n...
A confidence threshold is the cheapest thing that looks like a safety control, and it fails in the one place you need it: models are most confident on inputs they have neve...
Two wearables can look identical and be different products. One computes a quantity the sensing can actually resolve; the other infers a quantity it cannot, and covers the...
A workflow engine guarantees your process resumes. It does not guarantee your side effect happened once. Those are different properties solving different halves of the same...
Almost every agent platform sells the part that is easy and leaves you the part that is hard. The loop, the tool dispatch, and the trace viewer are commodity. Whether your...
This is usually argued on latency and cost, which are the two least decisive inputs. The questions that actually settle it are whether the system must work with no connecti...
The decision is not about quality and it is not close. Retrieval changes what the model knows; fine-tuning changes how it behaves. Almost every team asking this question ha...
An AI feature does not scale like the rest of your product. Its cost is per-request and does not amortise, its latency degrades in a shape most load tests never produce, an...
Cross-tenant leakage in a retrieval system is not an accident that happens to careless teams. It is the default outcome of a design that treats tenancy as a filter applied...
A 99 percent accurate detector is not a 99 percent safe robot, and the gap is not a matter of degree. Safety depends on which errors occur, how they are distributed in spac...
The perception team ships a detector that meets its accuracy target. The controls team builds a controller that is stable given its inputs. Both are right and the robot beh...
Prevention mechanisms fail silently. Reconciliation is the only part of a money-moving system that can tell you they did, and it is the component most often deferred, becau...
An idempotency key is not a header you add. It is a claim about what constitutes the same operation, and almost every double-charge traces to that claim being wrong: the ke...
When a unit leaves the building, the set of decisions you can still change shrinks to almost nothing. The design question becomes which properties must be right at manufact...
An over-the-air update is the only operation where a bug can remove your ability to fix the bug. That single property should determine the entire design, and it produces re...
Most engineering practice assumes you can stop the thing to work on it. Remove that and the entire method changes: you diagnose from passive observation, you change one var...
Infrastructure engineering solved a problem AI agents are currently rediscovering. Declaring the state you want, rather than the action you want taken, makes repeating an o...
A platform that works at a hundred nodes and fails at twenty-five thousand did not run out of capacity. It met failures that do not exist at small scale: monitoring that da...
The model is rarely the problem on a factory floor. The problem is the path between the controller and the model: a protocol implemented as the vendor felt like implementin...
Downtime on an automated line runs from hundreds of thousands to millions of dollars an hour. Convert that to a per-minute figure and it becomes an engineering budget, whic...
Your bench asks whether the signal looks right. A reviewer asks what the system does when it is wrong, across every unit you will ship, in conditions you did not choose. Th...
In most software, retraining is a Tuesday. In a regulated device it can be a regulatory event, and the cost is not the retraining but the clearance cycle it can trigger. Th...
An air gap is usually treated as a deployment constraint, something to solve at the end with a USB drive and a procedure. It is not. It invalidates assumptions at every sta...
Every convenience in a cloud AI stack is a form of retry: call again, fall back to a bigger model, log it and fix it tomorrow. Remove the network and all of them go at once...
Every component passes its own tests. Every team's investigation ends politely at their boundary. The system does not work. Five signs tell you the failure lives between co...
Seven questions, in the order that sorts candidates fastest. The first three separate people who have run a retrieval system in production from people who have read about o...
A diagnostic is not a discovery phase, an architecture review, or a slide deck of observations. It is a finding with evidence attached, a ranked list of what else is wrong,...
Monitoring answers whether the system is running. Evaluation answers whether it is right. Those are two different dials and almost every organisation has built only the fir...
There is a test that decides it, and it is not about model quality. If you cannot compute, in code, a number that tells you whether the task is closer to done, you cannot b...
The expensive failure is almost never a vendor who cannot execute. It is a competent team executing precisely, on the wrong component. That is much harder to detect, becaus...
You should not have to read a trace to know whether an agent is safe to ship. Five questions, none of them technical, each with a right answer and a recognisable wrong one....
Nobody in this field answers the cost question straight, so here it is. The price is not set by your system's complexity, it is set by which of four diagnoses you land on,...
Naming a pipeline stage 'verifier' does not make it one. If the stage is another model call asked whether the output looks right, it shares the generator's blind spots, pro...
Floyd gave us the conditions a terminating loop has to satisfy in 1967. Negate them one at a time and you get an exhaustive list of the ways termination can fail, which tur...
Your planner stops. Your executor stops. Both proofs are valid and neither is wrong. Put them in a loop together and the system runs forever, because termination is not a p...
TCP already had a bound before congestion collapse. The receive window was real, correctly enforced, and nobody had a bug in it. The internet fell over anyway, because that...
A ten-second check that tells you whether a variable is doing real work or is scaffolding: delete it, rerun, and see what changes. Delete TCP's congestion window and the in...
Two contractors quoted four and six months to rebuild a model. The actual fix took five days and was not the model. The decision between rebuilding and repairing is not a j...
An organisation with a hard cross-layer production problem weighs four options, and each is structurally wrong for it in a different way. The large integrator is broad and...
In 1990 Cohen and Levesque formally specified what it means for an agent to hold a goal: it believes the goal is not yet achieved, it wants it achieved, and it maintains th...
Nine agent frameworks, from nine organisations, in different languages, with no shared lineage. Every one decides whether to keep going using the same two ingredients: the...
A hung tool call freezes the iteration counter forever, so a max-iteration cap can never fire on a hang. Only a clock catches it, which is exactly why every embedded watchd...
The analog engineer designed a correct reference loop. The actuation engineer designed a correct current source with a safety limit. The mechanical engineer placed the elec...
Across a decade of systems in defense, medical devices, industrial automation, fintech, and infrastructure at fleet scale, the failure is almost never in the model. It is i...
Two current-limit resistors in series for redundancy is good design. The same two resistors, both the correct value, placed in parallel instead, quarters your limiting and...
When you measure off a body you are not measuring your signal. You are measuring your signal plus the person plus the room, all summed at the amplifier, and the thing you w...
The third electrode is a feedback loop wrapped around a person. It senses the body's common mode and drives against it, which is what turns a hope about electrode symmetry...
Munitions inspection in 1945, early stopping in clinical trials, Bayesian optimization, multi-armed bandits, index policies, and proof assistants that will not compile unti...
Whatever your sensor and that first little circuit throw away is gone. You cannot get it back later with software. A photo that comes out blurry does not become sharp with...
Two unglamorous techniques between a change and everyone. Shadow runs the new version against real traffic without showing anyone the results, so you find out how it behave...
Use the same model to write an answer and to judge it, and the loop closes on itself: it keeps the same blind spots twice, and it goes easy on itself in ways that are measu...
If the answer lives in a document your system never pulled up, nothing downstream can rescue it. The model cannot quote a page that was never opened, so it answers confiden...
Durable execution replays a workflow and skips the steps that already completed, which narrows the window in which a side effect can run twice. It does not close it, becaus...
Chess has a draw, which is not a loss. A solver returns unknown, which means it ran out of budget and not that no answer exists. A database names a deadlock victim so the t...
A real system had 11 internal steps and logged exactly one thing: did the whole run work, yes or no. So when it failed it looked random, and good engineers said it fails ra...
A chatbot writes words, and if it is wrong you delete it. An agent acts: it sends the email, charges the card, deletes the record, places the order, and those actions turn...
One step missed a single document. That assumption reached the planner, which generated three subtasks that each inherited it, and their outputs became the next step's inpu...
Four properties of every model are always present and almost never visible: it extrapolates past its data, it discards most of your input, it is memory-bound on a shared bu...
A system that is wrong costs you one answer. A system that is load-bearing and wrong costs you everything standing on that answer. In a trusted internal tool the output is...
You cannot read forty thousand answers, so you appoint one number as a messenger and send it to check the truth. Every dashboard number is that: a stand-in for something yo...
A test set drawn at random from your training data measures how much your test set resembles your training set, not whether the system works. Building one that predicts pro...
When an AI system works in testing and fails in production, the model is the last thing to suspect, not the first. Four checks rule out the plumbing before you touch anythi...
On a closed-loop bio device, the instant you turn the actuator on, the signal can vanish to a flat line, and it is not coupling. The current you inject is a loop that has t...
Building the agents is about 20% of the work. The 80% that decides whether an agent survives production is the part everyone skips: reconciliation when systems disagree, co...
An AI agent can run green for 264 hours and bill 47,000 dollars with every single call bounded, because the iteration cap is a fuel gauge, not a progress bar. It measures h...
A model does not read your input, it reads a compressed representation of it, keeping only the features it found useful and throwing the rest away. So cleaning up a messy i...
On fixed hardware, inference latency is set by moving bytes, not by doing math. Most models sit in the memory-bound region with the compute units idle, so the real cost is...
A model is geometry fit to data, so past the edge of its training it does not stop, it extrapolates, applying a flat rule that was anchored to nothing, with full confidence...
When an agent says it called a tool, the run comes back green, and nothing happened, the model almost never lied. A tool call is just text the model hands off, and it gets...
A wrong RAG answer is not one failure, it is two: the search pulled the wrong pages, or the search worked and the model ignored them. From the output they look identical, s...
When an agent charges a card twice or sends an email twice, nothing is broken to go find: no error, no red in the logs, the action was correct and simply ran twice. Duplica...
A RAG system that works today degrades quietly as its documents change. There are two kinds of drift: the easy one, where the questions start looking different, and the nas...
AI agents almost never fail because of the model. Chain enough steps and the math turns against you: at 95% reliability per step, a 20-step agent finishes under 40% of the...
A RAG system that scored 95% in testing and collapses in production almost never needs a model rebuild. The failure is usually in one corner of the pipeline, and the fix is...
When a model scores 95% in staging and collapses to 44% in production, the model is almost never the cause. The usual fix, rebuilding the model, retrains on the same data a...
On a wearable device, where the sensor physically sits is not a comfort decision, it is the decision that determines whether you need an AI model at all, and whether you sh...
For people building systems that get one shot:
edge, defense, medical, industrial.