Training Pipelines
Training pipelines automate the steps that turn versioned data into an evaluated model artifact. They are not just scheduled notebooks: they must make inputs, parameters, outputs, and approval evidence reproducible.
The pipeline steps
A pipeline typically extracts point-in-time data, validates schema, builds features, trains, evaluates, records an experiment tracking run, registers a candidate model version, and publishes a release report. It consumes dataset versioning manifests and often depends on upstream feature pipelines.
flowchart TD Data[Point-in-time data and dataset manifest] --> Validate[Validate schema and check label leakage] Validate --> Features[Build features] Features --> Train[Train model] Train --> Evaluate[Evaluate against release gates] Evaluate --> Register[Register candidate model version] Register --> Report[Publish release report]
Artifact: Pipeline DAG
dag: churn_training
schedule: "0 3 * * *"
tasks:
- id: build_dataset
outputs: ["dataset_manifest.yaml"]
- id: validate_schema
requires: [build_dataset]
fail_on: [missing_required_column, label_leakage_check]
- id: train_model
requires: [validate_schema]
outputs: ["model.pkl", "training_metrics.json"]
- id: evaluate_release
requires: [train_model]
gates:
validation_auc: ">= 0.84"
p95_latency_ms: "<= 120"
- id: register_candidate
requires: [evaluate_release]
outputs: ["model_registry_version"]CD for ML should test this DAG and block promotion when artifacts are missing. A completed run means the workflow executed, not that labels or business assumptions are correct.
Failure Modes
Pipelines fail semantically when point-in-time joins leak future data, when random seeds and splits change silently, or when a retry publishes a partial artifact. Make tasks idempotent and validate outputs before registration.
References
Nav