Production ML Architecture Patterns That Transfer Across Industries

The Realization That Changed How We Build

After shipping ML systems into production across eight different industries over four years, I had an uncomfortable realization: we kept solving the same problems from scratch.

A defect detection system for an automotive parts manufacturer. A predictive maintenance pipeline for an oil rig. A document classification system for a defense contractor. A biomarker detection model for a biotech startup. On the surface, these look like completely different projects. Different data modalities, different inference requirements, different regulatory constraints.

But when I mapped out the actual system architectures side by side, roughly 80% of the components were functionally identical. The same data ingestion patterns. The same feature store abstractions. The same model serving infrastructure. The same monitoring and alerting logic. The same retraining triggers.

The 20% that differs is crucial -- it is where domain expertise creates value. But the 80% that is shared? That is where most teams waste months reinventing solutions that have already been proven in other industries.

Here are the seven architecture patterns that transfer across every production ML deployment we have built.

Pattern 1: The Immutable Feature Pipeline

Every production ML system needs to transform raw data into features. The pattern that works universally is what I call the immutable feature pipeline: a DAG of deterministic transformations where every intermediate result is versioned, timestamped, and reproducible.

The implementation details vary. In manufacturing, your raw data might be sensor readings from PLCs arriving via MQTT. In fintech, it is transaction logs from a Kafka stream. In biotech, it is sequencing output files landing in S3. But the pipeline architecture is the same:

Raw Data → Validated Data → Base Features → Derived Features → Feature Store

Each stage is idempotent. Each stage writes to append-only storage. Each stage logs its transformation version so you can reproduce any historical feature set exactly.

We implement this with Apache Airflow for batch pipelines and a combination of Kafka Streams and Flink for real-time. The specific orchestrator matters less than the principles: immutability, reproducibility, and point-in-time correctness.

Point-in-time correctness deserves special emphasis. If you cannot answer the question "What features would this model have seen at inference time on March 15th at 2:47 PM?" with exact precision, you have a training-serving skew problem waiting to happen. Every industry hits this. Every industry underestimates it.

Pattern 2: The Shadow Deployment Gateway

Never cut over from an old system to a new ML model in a single deployment. This is true in manufacturing. It is true in defense. It is true in every domain we have worked in.

The pattern is a gateway layer that sits in front of your model serving infrastructure and supports four modes:

1. Shadow mode: New model receives live traffic, produces predictions, but predictions are logged only -- not served to downstream consumers
2. Canary mode: New model serves a configurable percentage of traffic (typically 1-5% initially) while old model handles the rest
3. A/B mode: Traffic is split deterministically by a partitioning key so you can measure per-segment performance
4. Full deployment: New model serves all traffic, old model remains warm for instant rollback

We implement this as a lightweight proxy service -- typically a FastAPI application behind an nginx reverse proxy -- that reads routing configuration from a feature flag service. The routing logic is simple. The value is enormous: it eliminates the "deploy and pray" pattern that causes most production ML incidents.

In our defense projects, this pattern is mandatory because the consequences of model regression are severe. In our biotech projects, it is mandatory because regulatory requirements demand documented evidence of model performance before full deployment. In manufacturing, it is mandatory because a bad model can halt a production line.

The universality of this need is remarkable. The gateway code is nearly identical across all our projects.

Pattern 3: The Three-Layer Monitoring Stack

Model monitoring is where most teams fail, and they fail in the same way regardless of industry: they monitor model accuracy but not the upstream conditions that cause accuracy to degrade.

Our monitoring stack has three layers:

Layer 1: Data quality monitoring. Before your model ever sees a data point, validate that the input data matches expected distributions. Schema validation, range checks, null rate monitoring, feature distribution drift detection using KL divergence or Population Stability Index. This layer catches 60% of production issues before they affect predictions.

Layer 2: Model behavior monitoring. Prediction distribution monitoring, latency tracking, throughput metrics, confidence score distributions. You do not need ground truth labels for this layer -- you are monitoring the model's behavior patterns and alerting on deviations from baseline.

Layer 3: Outcome monitoring. When ground truth eventually arrives (hours, days, or weeks later depending on domain), compute actual performance metrics and compare against thresholds. This is the layer everyone builds first, but it is the least useful for real-time issue detection because of the inherent label delay.

The implementation uses Prometheus for metrics collection, Grafana for dashboards, and custom Python services for statistical tests. We alert on Layer 1 issues with a five-minute SLA, Layer 2 with a fifteen-minute SLA, and Layer 3 on a daily digest.

This exact stack has been deployed in a fish processing plant, an oil refinery, a government agency, and a Series B startup. The dashboards look different. The thresholds are calibrated differently. The architecture is the same.

Pattern 4: The Feedback Loop Accelerator

Every production ML system needs to get better over time. The pattern for achieving this is an active learning loop that prioritizes which new data points to label based on model uncertainty.

The implementation:

1. Model produces predictions with calibrated confidence scores
2. Low-confidence predictions are routed to a human review queue
3. Human reviewers label the uncertain cases (this is where domain expertise gets encoded)
4. Labeled data is added to the training set
5. Model is retrained on a schedule (weekly for most applications, daily for high-velocity domains)
6. New model goes through the shadow deployment gateway before replacing the current model

The key insight that transfers across industries: the human review interface must be designed for the domain expert, not the ML engineer. In manufacturing, that means an interface where a quality inspector can label defect images with three clicks, not a general-purpose annotation tool that requires a tutorial. In oil and gas, that means a dashboard where a process engineer can confirm or reject anomaly flags in the context of the full sensor telemetry.

We have built custom labeling interfaces for every major project. The backend -- the queue management, the active learning selection logic, the data versioning -- is the same across all of them. The frontend is always custom.

Pattern 5: The Configuration-Driven Model Registry

Model versioning is a solved problem in the abstract. MLflow, Weights & Biases, and a dozen other tools handle it well. The pattern that matters in production is not just tracking models but encoding the entire serving configuration as a versioned artifact.

A serving configuration includes:

• Model weights and architecture
• Preprocessing pipeline version (the exact feature transformations)
• Postprocessing logic (thresholds, business rules, output formatting)
• Hardware requirements (GPU type, memory, batch size limits)
• SLA requirements (latency p99, throughput minimum)
• Rollback target (which previous configuration to revert to on failure)

We store this as a YAML manifest in the model registry alongside the model artifacts. When a deployment happens, the deployment system reads this manifest and configures the serving infrastructure accordingly. This eliminates the "it works on my machine" class of deployment failures and the more insidious "the model is fine but the preprocessing changed" class of bugs.

Pattern 6: The Circuit Breaker Fallback

What happens when your model fails? Not degrades -- fails. The serving container OOMs. The GPU throws a hardware error. The feature store is unreachable.

Every production system needs a circuit breaker pattern: when the ML system fails or exceeds latency SLAs, traffic is automatically routed to a fallback. The fallback hierarchy we use:

1. Cached prediction: If we have seen this exact input recently, serve the cached result
2. Simple model fallback: A lightweight model (logistic regression, decision tree) that can run on CPU with no external dependencies
3. Rule-based fallback: Hand-coded business rules that approximate model behavior for the most common cases
4. Graceful degradation: Return a "low confidence" flag and let the downstream system handle it (usually by routing to human review)

In manufacturing, the fallback might be "flag this part for manual inspection." In a recommendation system, it might be "serve the most popular items." In a defense application, it might be "default to the most conservative classification."

The circuit breaker logic itself is identical: monitor error rates and latency over a sliding window, trip the breaker when thresholds are exceeded, attempt recovery after a configurable cooldown period.

Pattern 7: The Reproducibility Contract

Every ML system we deploy comes with what we call a reproducibility contract: a guarantee that any historical prediction can be reproduced exactly given the same inputs.

This requires:

• Data versioning: Every training dataset is immutable and versioned (DVC or a custom solution on top of cloud object storage)
• Code versioning: Every model training run is tied to a specific git commit
• Environment versioning: Docker images for training and serving are immutable and tagged
• Configuration versioning: All hyperparameters, feature engineering parameters, and serving configurations are stored in version control
• Random seed management: All stochastic processes use documented seeds

The reproducibility contract is table stakes in regulated industries (defense, healthcare, finance). But we enforce it everywhere because it is the foundation that makes debugging production issues tractable. When a model produces an unexpected prediction, you need to be able to trace back through the entire pipeline -- from raw data to features to model version to serving configuration -- and understand exactly why.

The 20% That Must Be Customized

These seven patterns form the backbone. Here is what differs:

Latency requirements range from sub-10ms (real-time bidding, robotic control) to minutes (batch document processing). This drives the choice between online serving (TorchServe, Triton) and batch inference (Spark, custom batch jobs).

Regulatory constraints vary wildly. ITAR for defense. HIPAA for healthcare. GMP for pharmaceutical manufacturing. Each imposes specific requirements on data handling, model documentation, and deployment processes.

Data modalities determine the feature engineering approach. Time series from industrial sensors need different preprocessing than medical images, which need different preprocessing than natural language documents.

Feedback loop latency ranges from seconds (online learning in ad tech) to months (clinical trial outcomes in pharma). This determines the retraining cadence and the relative importance of the three monitoring layers.

Building Your Own Pattern Library

The highest-leverage investment an ML engineering team can make is building internal pattern libraries that encode these reusable architectures as templates, cookiecutters, or internal platforms.

At Opulion, our internal platform lets us spin up a new production ML project with all seven patterns pre-configured in under a day. The first week of any engagement is spent customizing the 20% -- calibrating monitoring thresholds, building domain-specific labeling interfaces, tuning the feature pipeline for the specific data modality.

This is why we can deliver production ML systems in weeks rather than months. Not because we are smarter than anyone else. Because we have done the work of identifying what transfers and encoding it into reusable infrastructure.

The model is the least interesting part. The patterns are where the value lives.