ML Engineering and MLOps

ML engineering and MLOps covers the lifecycle around a model: versioned data, reproducible training, evaluation gates, deployment, monitoring, reliability, and incident response. The model is only one artifact. A production ML system also includes datasets, feature definitions, pipelines, serving contracts, dashboards, rollback paths, and ownership.

Use this section when the question is “how do we operate this model safely and repeatedly?” For infrastructure primitives, see Cloud and Distributed Systems; for experiment design and release evidence, see Experimentation and Evaluation.

Knowledge map

The lifecycle runs from training and versioning through serving, staged release, operation, and drift response, then loops back when monitoring finds new evidence.

flowchart TD
  Lifecycle[ML System Lifecycle] --> Train[Training Pipelines and CI/CD]
  Train --> Track[Experiment, Dataset, Model Versioning]
  Track --> Serve[Model Serving and Inference]
  Serve --> Release[Shadow, Canary, Rollbacks]
  Release --> Operate[Monitoring, Observability, SLOs]
  Operate --> Drift[Drift, Degradation, Incident Response]
  Drift --> Lifecycle

Reading path

Read the lifecycle and training first, then artifacts, serving, release, operation, and drift.

1. ML System Lifecycle: the end-to-end path from framing to retirement.
2. Training Pipelines: automated, auditable model production.
3. CD for ML: tests and promotion gates for code, data, and models.
4. Experiment Tracking: recording the evidence behind runs.
5. Dataset Versioning: immutable, referenceable training data.
6. Model Versioning: registering and promoting model artifacts.
7. Evaluation Datasets: the held-out sets that gate release.
8. Golden Datasets: curated regression sets for critical behavior.
9. Model Serving: the runtime layer for reliable inference.
10. Batch and Online Inference: scheduled versus request-time scoring.
11. Microservices: decomposing the prediction path into services.
12. Docker: reproducible deployment units.
13. Shadow Deployment: testing on copied traffic without exposure.
14. Canary Deployment: progressive exposure with guardrails.
15. Rollbacks: reverting safely when a release goes wrong.
16. A-B Testing: controlled comparison of model variants.
17. Monitoring: tracking health and quality in production.
18. Observability: traces and logs that explain behavior.
19. Service Level Objectives: the reliability targets to hold.
20. Reliability: timeouts, retries, and graceful degradation.
21. Data Drift: input distributions moving away from training.
22. Concept Drift: the input-output relationship changing.
23. Model Degradation: quality decay without label change.
24. Production Incident Response: handling model incidents.
25. Human-in-the-Loop Systems: routing uncertain cases to people.
26. Active Learning: choosing which examples to label next.

Connections

• Data Engineering supplies the pipelines and features these systems consume.
• Cloud and Distributed Systems runs the infrastructure, and Responsible AI governs deployed behavior.