A Rule With No Instance Is Not Found by Inspection

Mostafa Dhouib·8 min read

A Rule With No Instance Is Not Found by Inspection

The short answer. A rule that only fires on a condition absent from your copied data cannot be found by reading the schema, no matter how carefully you read it. You can catalogue the branch, state what it would do, and still have no idea how often it is taken or what it produces on real inputs, because nothing in your copy exercises it. That is a structural limit of static discovery rather than a diligence failure. The remedy is not a better catalogue. It is keeping the old system running while the new one is built, sending live traffic through both, and comparing outputs until divergences stop appearing. Completeness comes from traffic, not from reading.

There is a comfortable belief at the end of a thorough discovery phase, which is that the hard part is now over.

The catalogue is complete. Every procedure, trigger and constraint is enumerated and classified. Someone can describe the organisation's rules in writing for the first time in a decade. The temptation to treat that document as the specification is strong, and acting on it is how otherwise careful migrations fail.

What inspection can and cannot see

Static discovery gives you existence. It does not give you incidence.

Reading a procedure tells you there is a branch for a particular customer type, a particular currency, or a particular combination of status and date. It tells you the condition and the consequence. What it cannot tell you is whether that condition occurs in practice, how often, in what distribution, and in combination with what else.

That gap matters because the consequence of a rule depends on its inputs, and the inputs you have are the ones in your copy.

Three concrete versions of the problem.

The annual branch. Policy that only applies at financial year end. If your copy is a point-in-time snapshot taken in the middle of a year, nothing in it has ever taken that path. You will read the code and conclude you understand it, and understanding written logic is not the same as having observed it run.

The absent category. A rule for a customer class that currently has no members, a currency nobody has transacted in for two years, a product line that was retired but not removed. The branch is live code with zero instances, and the first instance after cutover is a production incident.

The combination. Two rules that each behave as documented and interact in a way neither description mentions. Inspection examines objects one at a time, which is the only tractable way to examine them, and interaction is by definition not visible object by object.

None of these are findable by reading harder. Reading harder finds more branches, which is useful, and it does not convert a branch into an observation.

Why this reorders the project

The usual mental model is that discovery reduces risk and then the build consumes the remaining risk. On that model, a more complete catalogue means a safer cutover, so effort should go into the catalogue.

The real relationship is different. The catalogue reduces the volume of surprises. The continued existence of the old system determines whether a surprise is survivable. Those are independent, and only one of them has an upper bound you can reach.

This is why the decision that matters most is not how thorough discovery was. It is whether the old system is still running on the day the new one starts taking real work.

Shadow running

Shadow running means the new implementation receives real traffic and produces real outputs, while the old implementation remains the one whose outputs are used.

The mechanics are simple and the discipline is not.

Duplicate the input, not the effect. The new path must see the same request the old path saw. It must not send the email, charge the card, write to the integration or update the shared record. Separating read behaviour from write behaviour cleanly enough to run a shadow is usually the single largest piece of engineering in the exercise, and discovering that late is a common way for shadow running to get dropped from scope.

Capture both outputs in full. Not a checksum. The actual values, so that a divergence can be diagnosed rather than merely counted. A difference you can see is a finding. A difference you can only count is an anxiety.

Compare on business meaning, not on bytes. Two outputs can differ in field ordering, whitespace, timestamp precision or identifier allocation and be identical in substance. If the comparator does not normalise those, every record diverges and the signal disappears into noise on the first day.

Keep a divergence log with dispositions. Every divergence is either a bug in the new path, a known and accepted difference, or a rule nobody had catalogued. That third category is what shadow running is for, and it is the only mechanism that produces it.

Deciding when a path is done

The question shadow running has to answer is when to switch a path over, and the answer is not a date.

A workable rule has three parts. The path has run in shadow across a period that includes its known cyclical events, such as a month end if the path has month end behaviour. New unexplained divergences have stopped appearing for a defined stretch of real traffic rather than a defined stretch of calendar time. And every divergence in the log has a disposition, with the accepted ones accepted by someone whose job it is to accept them rather than by the team that wrote the code.

Volume matters more than duration. A path that handles a thousand transactions a day reaches confidence in a week. A path that handles four reaches the same confidence in a year, and pretending otherwise because the calendar looks the same is the most common error here.

The rule you will still miss

Shadow running finds rules that fire during the shadow period. A rule that fires once every three years is not found by a three month shadow, and the honest position is that it will be found in production.

What changes is the cost of finding it. With the old system retired, that discovery is an outage with no fallback and a growing body of new data that cannot be reverted. With the old system still present and the path individually reversible, it is a bad day. The engineering goal is not to eliminate the discovery. It is to make sure the discovery happens while the answer is still available.

The honest limit

Shadow running is expensive and it is not always available.

It requires the old system to keep running, which means paying for it, maintaining it, and having staff who can still operate it. It requires the new path to be side-effect free in shadow mode, which is intrusive and sometimes impossible. Where the old system's behaviour depends on writing to a shared external system, there may be no clean way to run both without doubling real-world effects.

It also costs time in a project that is usually late already, and the pressure to shorten the shadow window is relentless. A shortened window is not worthless, and it should be shortened by naming which cyclical events you are now choosing not to observe rather than by moving a date.

Finally, comparison only catches what the comparator looks at. A divergence in a field nobody compared is invisible, and deciding which fields are load-bearing is a judgement made by the same people who might have missed the rule in the first place.

FAQ

Why can a thorough schema review still miss a business rule? Because inspection gives you existence and not incidence. Reading a procedure shows a branch and its condition, but if no record in your copied data satisfies that condition, nothing exercises the branch. A rule for a retired product line, an empty customer class or a financial year end absent from a mid-year snapshot is catalogued accurately and never observed.

What matters more, a complete catalogue or keeping the old system running? Keeping the old system running. The catalogue reduces the number of surprises and has an upper bound you can actually reach. The old system's continued presence determines whether a surprise is a bad day or an unrecoverable outage, and that is the variable with the larger effect on the outcome.

What is shadow running in a migration? Sending real production traffic to both the old and the new implementation while only the old one's output is used. The new path must receive the same input but produce none of the side effects, so no messages are sent and no shared records are written. Both outputs are captured in full and compared so that divergences can be diagnosed rather than merely counted.

How do you compare outputs between an old and a new system? On business meaning rather than on bytes. Field ordering, whitespace, timestamp precision and identifier allocation routinely differ without any substantive difference, so the comparator must normalise them or every record diverges and the real signal disappears. Then maintain a divergence log where each entry is dispositioned as a bug, an accepted difference, or a previously uncatalogued rule.

When is a path ready to switch over? When it has run in shadow across its known cyclical events, when new unexplained divergences have stopped appearing over a meaningful volume of traffic rather than a meaningful stretch of calendar, and when every logged divergence has been dispositioned by someone whose job it is to accept the risk. Volume matters more than duration, because a low-traffic path accumulates evidence slowly.

What does shadow running still fail to find? Any rule that does not fire during the shadow window. A rule triggered once every few years will not appear in a three month shadow, and it will eventually be discovered in production. What shadow running changes is the cost of that discovery, because the old system is still available to answer what the correct behaviour was.