DevOps to MLOps: Building the Shared Delivery Muscle

The new variable

Code + Config w, code + config always produces the same artifact deterministically. Add a training dataset and that guarantee breaks. Two identical codebases trained on different data snapshots produce different models. That single fact explains why every DevOps practice needs a data-aware twin in MLOps. Visualize , IaC, observability) and the MLOps circle (data versioning, model registry, drift monitoring) share a large intersection — but the non-overlapping parts are where teams get surprised.

• What DevOps gives you free Inherited: Version control, CI pipelines, containerization, blue/green deploys, and observability dashboards — all transfer unchanged to model serving. Use them; do not reinvent them.
• What MLOps adds Extended: Data versioning, feature stores, experiment tracking, model registries, and drift-triggered retraining pipelines. Bolt these onto existing CI/CD; they extend the loop, not replace it.
• Where teams get surprised Risk: Models degrade silently without retraining, fail on unseen distributions, and encode training-set biases. Unlike a crashing service, a drifting model gives no stack trace — only slow KPI decline.

CALMS applied to ML

Five pillars — all five have direct ML analogs

CALMS is DevOps culture in an acronym. Animate , pipeline-as-code for data and model workflows, automated evaluation gates. Every step reproducible from a single git SHA. Animate the pipeline DAG lighting up green , lead time, change failure rate, MTTR) alongside ML-specific metrics (PSI, accuracy drift, feature freshness). One dashboard, both signal families.

• Sharing Enablement: Model cards communicate assumptions, limitations, and known failure modes to downstream consumers. Internal demo days and shared registries prevent duplicate work and silent assumptions.

Four DORA metrics, ML edition

Track the same four metrics for both artifacts: code and model

DORA metrics were validated across 33,000 software teams — they apply directly to ML. The key insight: track them for two parallel pipelines and overlay them on one dashboard. The bottleneck is always visible , the right lane is model deploys, narrowing together toward production.

1. Deployment Frequency (Goal: weekly+): How often do you ship a new model version to production? Elite ML teams retrain weekly or on drift events. Visualize ): From dataset snapshot to trained artifact to deployed endpoint. Cached feature stores, containerized trainers, and evaluation automation compress this to hours. Color-code each stage ): Percentage of model deployments that degrade KPIs or trigger a rollback. Automated quality gates checking PSI, AUC delta, and P99 latency keep this below 10 %. Chart ): How quickly does the team recover after a bad model deploy? Blue/green routing and a warm fallback model (serving 5 % of traffic at all times) cut MTTR from hours to minutes.

Reliability in numbers

Put concrete targets on what "reliable" means

Engineering SLO conversations improve dramatically when targets are written ) and P95 (long tail). P95 spikes reveal data validation failures or resource contention — the usual culprits in ML pipelines.

$$ \text{MTTR} = \frac{1}{n}\sum_{i=1}^{n}\left(t_{\text{restore},i} - t_{\text{incident},i}\right) $$

ML MTTR often inflates because teams must diagnose whether the failure is data, model, or infrastructure before they can fix it. Runbooks with decision trees cut this time in half.

• Run weekly error-budget reviews with separate budgets for software SLOs and model quality SLOs.
• Maintain a warm fallback model serving 5 % of traffic — rollback becomes a router config change, not a redeploy.
• Write incident runbooks ] Conduct quarterly DR drills: simulate losing the feature store, the model registry, and the training cluster sequentially.

Reference architecture

Three planes, clear ownership, explicit interfaces

Structure the shared DevOps/MLOps stack into three planes — each with a distinct owner and a versioned interface to the others. Animate , revealing team boundaries and data-flow arrows between planes.

Data Plane: Owned by Data Engineering. Ingestion pipelines, feature store, labeling queues, quality monitors. Interface to the Model Plane: a versioned feature API with freshness SLA — visualize , experiment tracker, evaluation harness, model registry. Interface to the Delivery Plane: a versioned model artifact + evaluation report card. Animate lineage arrows flowing upward.

Delivery Plane: Owned by Platform/SRE. CI/CD, canary router, serving infrastructure, observability mesh. Interface back to Data and Model Planes: a drift alert stream that automatically triggers remediation jobs.

Related posts:

• mlops-systems-blueprint
• data-warehouse-lake-lakehouse
• neural-architectures-visual-guide