Production Incident Response

Production incident response coordinates detection, triage, mitigation, communication, and learning when a live ML system behaves unsafely or unreliably. ML incidents include ordinary outages, but also wrong model behavior with green infrastructure metrics.

The response loop

The response loop is declare, assign roles, stabilize, preserve evidence, communicate, mitigate, and run a blameless review after recovery. During the incident, restoring acceptable behavior matters more than proving the root cause. Mitigation may mean rollbacks, disabling automation, routing to human-in-the-loop systems, freezing retraining, or lowering traffic.

Artifact: Incident Runbook Contract

incident_type: model_behavior_degradation
severity: SEV2
declare_when:
  - "primary decision SLO violated for 15m"
  - "manual review queue exceeds 2x normal volume"
roles:
  incident_lead: ml-platform-oncall
  comms: product-ops
  data_owner: risk-data-eng
first_15_minutes:
  - freeze canary ramps
  - snapshot model, feature, and threshold versions
  - compare affected traffic by model_version and segment
  - choose rollback or manual-review fallback
evidence_to_preserve:
  - prediction events
  - feature freshness metrics
  - model registry entry
  - deployment revision

This contract depends on monitoring for detection and observability for diagnosis. If those signals were not emitted before launch, responders will spend the incident reconstructing state.

Failure Modes

Common failures are unclear authority, delayed customer communication, overwritten logs, and postmortems that stop at “bad model” instead of tracing data, code, review, and rollout controls. Reliability work is complete only when follow-up actions are owned and tracked.

References

• Google SRE Book: Managing Incidents
• NIST AI Risk Management Framework

Nav

Section — ML Engineering and MLOps

← Model Degradation Human-in-the-Loop Systems →

Learning path — Production ML

← Monitoring