Monitoring

Monitoring watches known signals and alerts when they violate a defined expectation. ML monitoring must cover service health, data health, model behavior, and delayed outcomes. A model API can be up while stale features or shifted populations make its decisions wrong.

Four layers of monitoring signals

Good monitors name the metric, owner, aggregation window, threshold, labels, and action. ML monitoring stacks four layers, from signals that fire before any harm (leading) to ones that confirm harm only after labels arrive (lagging):

Layer	Example signals	Detects	Typical action
Service	latency, error rate, saturation, dependency failures	outages, overload	page on-call
Data	feature freshness, missingness, schema drift, range checks	broken inputs	block or fall back
Model	score distribution, prediction mix, drift proxies, fallback rate	silent behavior change	investigate, compare canary
Outcome	delayed labels, business KPIs, complaint rate	real quality loss	retrain or roll back

The gap between the fast leading layers and the slow outcome layer is the whole reason ML needs more than uptime dashboards: a model can be perfectly available while data drift or concept drift quietly degrade decisions, and the confirming labels (model degradation) may arrive days later.

Artifact: Prometheus Alert

groups:
  - name: fraud-model
    rules:
      - alert: FraudScorerHighFallbackRate
        expr: |
          sum(rate(fraud_predictions_total{fallback="true"}[10m]))
          /
          sum(rate(fraud_predictions_total[10m])) > 0.01
        for: 15m
        labels:
          severity: page
          service: fraud-scorer
        annotations:
          summary: "Fallback rate above 1% for fraud scorer"
          runbook: "https://runbooks.example.com/fraud-scorer/fallback-rate"

The alert is useful because it points to a specific owner and runbook. A dashboard without an action path belongs more to observability than paging. For forecasts, monitoring has extra temporal concerns such as horizon-specific error and freshness, covered in forecast monitoring.

Failure Modes

Monitoring fails when thresholds are copied across models, labels are missing from metrics, or alerts fire on symptoms nobody can mitigate. It also fails when it excludes model version and feature version, making production incident response reconstruct state from logs after the fact.

References

• Prometheus alerting rules
• OpenTelemetry Signals
• Google Rules of Machine Learning

Nav

Section — ML Engineering and MLOps

← MLOps A-B Testing Observability →

Learning path — Production ML

← Model Serving Production Incident Response →