Golden Datasets
Golden datasets are small, trusted, versioned examples that serve as acceptance references for model, prompt, retrieval, or pipeline changes. They complement larger evaluation datasets: a golden set is meant to be inspected and defended case by case.
What a golden case holds
A golden dataset should include inputs, expected outputs or grading criteria, slice tags, risk labels, source, reviewer, and version. It belongs in ci-cd-for-ml because regressions on known critical cases should block promotion even when aggregate metrics improve.
Cases are curated, not sampled: each one is drawn from a real incident, a legal or safety requirement, or a decision that must not silently change. The set stays small enough — tens to low hundreds of cases — that a reviewer can defend every entry. When a golden expectation genuinely needs to change, the correct move is a new version, never an in-place edit, so the history of what “correct” meant is preserved.
Golden versus evaluation sets
Golden and evaluation datasets answer different questions and fail in different ways:
| Aspect | Golden set | Evaluation dataset |
|---|---|---|
| Size | tens to low hundreds | thousands or more |
| Curated from | real incidents and must-not-regress decisions | representative sampling |
| Purpose | acceptance gate on critical cases | estimate aggregate quality |
| A failure means | block the release and inspect the case | a shift in an averaged metric |
| Reviewed | case by case, by a named owner | in aggregate |
Artifact: Golden Record Schema
golden_case:
id: "fraud-gold-0142"
input_ref: "s3://golden/fraud/0142.json"
expected:
decision: manual_review
min_score: 0.82
required_reason_codes: [new_device, velocity]
tags: [new_account, card_not_present, prior_incident_pattern]
source: "2026-05 incident review"
reviewer: "risk-ops"
version: "golden-fraud:v4"The experimentation section has the canonical page on golden datasets. This MLOps page emphasizes release blocking, ownership, and dataset versioning.
Failure Modes
Golden sets become brittle when expected outputs are overspecified for cases with legitimate ambiguity. They become useless when teams add only easy examples or silently edit expected labels after a model fails. Use human-in-the-loop systems to review contested cases and create a new version instead of mutating history.
References
Nav