If You Cannot Rebuild It, It Is A Source

Mostafa Dhouib·11 min read

If You Cannot Rebuild It, It Is A Source

The short answer. A three-layer record keeps an append-only source layer, a governed derived layer, and a revisable current-state layer that carries its own history. The rule holding the three together is that the derived layer must be regenerable from the source. Drop it, run the derivation again, and you should get the same thing back. The moment that stops being true, the derived layer has quietly become a source in its own right. Nothing in your governing documents describes it as one, so no change control, no retention rule and no audit obligation applies to it. Test for rebuildability deliberately. A layer that fails the test gets fixed, or gets promoted to source with the controls that status carries.

Most regulated systems I am asked to review have the three layers. Someone read about event sourcing, drew the diagram, and built it. What they rarely have is a test that proves the middle layer is still derived, and in the cases I have looked at closely, at least one derived table had stopped being derivable months earlier without anyone noticing.

The three layers, and the single rule between them

The shape is worth stating plainly, because the rule only makes sense once the layers are distinct.

The source layer is append-only. Facts arrive, get written, and are never edited. A correction is a new fact that supersedes an old one, not an update to the old row. This layer is the thing a regulator, an auditor or a court is actually interested in, because it is the only layer where "what did the system know, and when" has a defensible answer.

The derived layer is governed. It holds the computed consequences of the source facts: aggregates, projections, scores, normalised views, the shapes the application actually queries. It is governed in the sense that its definition is specified and its changes are reviewed, because changing a derivation changes the meaning of every record it produced.

The current-state layer is revisable, with history. This is where mutable working data lives. A user edits a profile, a clinician amends a note, an operator corrects a unit. The row changes, and the change itself is recorded, so you can reconstruct any earlier state.

The rule between the first two layers is the whole architecture. The derived layer must be regenerable from the source layer. That is not a performance claim or a nice property. It is the reason the derived layer can be revised at all. If the derivation definition was wrong for six months, you fix the definition and rebuild, and the record is correct again, because the source was never touched. Without that property, a wrong derivation is permanent damage.

How to test a derived layer for rebuildability

The test is not clever. It is just rarely run.

Rebuild into a shadow copy, never over the live one. Create a parallel set of derived tables, run the derivation from the source into them, and compare. Rebuilding in place destroys the evidence you were trying to gather and makes the test too frightening to run often, which means it never runs.

Rebuild from the beginning of the source, not from the last checkpoint. Incremental rebuilds test the increment. They say nothing about whether the derivation of a record written two years ago still works, which is exactly the record someone will ask about.

Compare content, not counts. Matching row counts is the most common false pass. Compare field by field, with an explicit list of the fields you intend to ignore, such as the timestamp recording when derivation ran. Everything not on that list must match.

Run it on a schedule, and treat a mismatch as a defect. A rebuild you run when someone asks is a demonstration. A rebuild that runs weekly and fails loudly is a control. The point of the schedule is that it catches drift close to the change that caused it, while the person who made the change still remembers making it.

Keep the derivation code versioned, and record which version produced each batch. A rebuild that uses today's code to reproduce last year's output is answering a different question. Both questions are useful. Only one of them is the question of whether the old records are still derivable, and you cannot ask it at all if the old derivation code no longer exists.

The four ways the test fails

When a rebuild does not reproduce the derived layer, the cause is almost always one of four things.

The derivation reads something that is not in the source layer. A configuration value, a feature flag, a threshold in an environment variable, a lookup table maintained by hand, a row from the current-state layer. The derivation is a function of the source plus that hidden input, and the hidden input has changed. This is the most common failure and the easiest to miss, because the code looks pure.

The derivation is not a deterministic function of its inputs. It reads the clock. It depends on the order rows come back from an unordered query. It accumulates floating-point values in whatever order the partitioning happened to produce. It generates identifiers that are not derived from the source. Each of these produces output that differs run to run in ways that are individually small and collectively fatal to the comparison.

Something wrote to the derived layer directly. A support tool. A one-off correction script. A migration. Someone fixed a broken total at two in the morning, which was the right call operationally, and in doing so created a value that the source cannot explain. This is the failure that turns a derived layer into a source most abruptly, because the information in that write exists nowhere else.

The derivation changed and the old definition was not kept. The code was refactored, the semantics shifted slightly, and the records produced before the change cannot be reproduced by the code after it. This is not a bug in the usual sense. It is an unrecorded change of meaning.

What to do when a layer fails the test

There are two honest responses and one dishonest one.

Fix it, so the rule holds again. Move the hidden input into the source layer, so that the configuration in force at the time is itself a recorded fact. Remove the nondeterminism by imposing an explicit sort, pinning the accumulation order, and deriving identifiers from source content. Close the direct write path and replace it with a mechanism that writes a correcting fact to the source and lets the derivation pick it up. Version the derivation and keep old versions runnable.

Promote it, and govern it as a source. Sometimes the dependency is real and cannot be removed. The derivation genuinely needs an input that only existed at the moment it ran, and you have no way to reconstruct it. Fine. Then that layer is a source. Say so. Make it append-only, bring it under change control, give it a retention rule and an audit trail, and name it in the governing document that describes your record. Promotion is not a defeat. It is an accurate description replacing an inaccurate one.

The dishonest response is to keep calling it derived. This is what happens by default, and it is worse than either alternative, because the architecture diagram now actively misleads the people relying on it. Your change-control process does not protect that table. Your retention policy may not cover it. Your recovery plan assumes it can be regenerated, and it cannot. Someone will eventually drop it to rebuild it, exactly as the design says they can, and the data will be gone.

Why this matters more as derivation gets cheaper

For most of the history of this pattern, a derived layer was expensive to create. Someone wrote the transformation, someone reviewed it, and the cost of that work kept the number of derived layers small and the attention paid to each one high.

That constraint is gone. A derived layer can now be a short pipeline someone generated in an afternoon, or a single call to a hosted model that turns free text into structured fields. Teams add derived layers faster than they can govern them, and the review that would have caught a hidden input never happens, because nothing about the change looked big enough to need it.

Model-based derivation is the sharp case, and it fails the rebuildability test by construction rather than by accident. A model call in the derivation path cannot be replayed to the same output, which is the subject of its own piece on why the lowest temperature setting is not determinism. The architectural consequence is the one described here. A model-derived layer is a source layer, and the correct move is to store the output and the model version as part of the record rather than planning to recompute it.

The habit worth building is smaller than any of this. In a design review, point at each table in turn and ask one question. What rebuilds this? If nobody can name the process, you are looking at a source that nothing governs. The decision one level up is which system holds the official copy.

The honest limit

Rebuild comparison is a reproducibility test, not a correctness test. A derivation can be reproducibly wrong, and this test will pass every time while the records stay wrong. It tells you that you can fix a bad definition and rebuild. It does not tell you the definition is good.

Exact equality is also the wrong bar in more cases than people expect. Derivation timestamps, surrogate keys and batch identifiers will differ legitimately. The discipline is to write the ignore list explicitly and keep it short, because an ignore list that grows whenever the comparison fails is a comparison that no longer checks anything.

Some derived layers depend on genuinely time-bound external inputs, such as a terminology version or a published rate in force on a given day. The answer there is to capture that input as a source fact at the time of derivation, not to classify the layer as unrebuildable and move on.

Finally, whether your record carries a formal obligation for the source layer, and what retention and audit trail that implies, depends on your regulatory scope and on the version of the governing rules that apply to you. Re-verify those against current text when you design the record. The architectural rule stated here holds in any regime. The specific obligations attaching to a layer you promote to source do not, and you need to go and read them.

FAQ

What makes a data layer a source rather than a derived layer? Whether it can be regenerated. If you can delete the layer, run its derivation from the append-only source, and get the same content back, it is derived. If you cannot, the layer holds information that exists nowhere else, which makes it a source regardless of what the architecture diagram calls it.

How do I test whether my derived layer is still rebuildable? Rebuild it from the start of the source into a shadow copy, then compare field by field with an explicit short list of fields you intend to ignore, such as the derivation timestamp. Run this on a schedule rather than on request, and treat any mismatch outside the ignore list as a defect to diagnose.

Why does it matter if a derived layer has become a source? Because nothing controls changes to it. Source layers get change control, retention rules and audit trails because a governing document names them. A layer everyone believes is derived gets none of that, and the recovery plan assumes it can be regenerated. Someone will eventually drop it to rebuild it and the data will not come back.

What are the usual reasons a rebuild does not reproduce the original? Four causes cover nearly all of it. The derivation reads an input outside the source, such as a configuration value or a flag. The derivation is nondeterministic, through clock reads or unordered accumulation. Something wrote to the derived layer directly, usually a support fix. Or the derivation code changed and the earlier version was not kept.

Can I just accept that a layer is not rebuildable? Yes, if you promote it honestly. Declare it a source, make it append-only, put it under change control, give it retention and audit handling, and name it in the governing document. What does not work is continuing to describe it as derived, because that leaves every process depending on the distinction operating on a false premise.

Why is this a bigger problem now than it used to be? Derivation used to be expensive, which kept the number of derived layers small and the review of each one serious. A derived layer can now be a pipeline built in an afternoon or a single model call, so they get added faster than they get governed. Model-derived layers fail rebuildability by construction, which makes them sources from the day they ship.