Experimentation and Evaluation

Experimentation and evaluation turns model behavior into evidence. Offline evaluation estimates behavior before launch, online experiments measure user or business impact under real traffic, and human or model-assisted review handles cases where there is no simple label. The core question is always the same: what decision will this evidence support, and what failure modes would the aggregate metric hide?

This section connects statistical testing from Probability and Statistics, forecasting backtests from Time-Series Forecasting, and production monitoring from ML Engineering and MLOps.

Knowledge map

Offline evaluation and its statistics feed online experiments; human and model-assisted evaluation handles subjective outputs; risk-aware measures guard safety-sensitive systems.

flowchart TD
  Offline[Offline Evaluation and Golden Datasets] --> Stats[Statistical Significance and Resampling]
  Stats --> Online[Online Experiments and A-B Testing]
  Offline --> Human[Human Evaluation and LLM-as-Judge]
  Human --> GenAI[Comparing Generative and Classical Systems]
  Online --> Risk[Coverage, Abstention, Risk-Weighted Errors]
  GenAI --> Risk

Reading path

Read offline evaluation and its statistics, then online experiments, subjective evaluation, and risk-aware measures.

1. Offline Evaluation: estimating quality before launch.
2. Golden Datasets: curated sets that gate critical behavior.
3. Calibration: whether predicted probabilities match reality.
4. Statistical Significance: distinguishing signal from noise.
5. Repeated Sampling: bootstrap estimates of uncertainty.
6. Paired Evaluation: comparing systems on the same items.
7. Online Experiments: measuring impact under real traffic.
8. A-B Testing: controlled randomized comparison.
9. Human Evaluation: rubric-based judgment of outputs.
10. LLM-as-Judge: using models to score generations.
11. Comparing Generative AI and Classical ML Systems: evaluating fundamentally different systems.
12. Coverage: how much of the input space is answered.
13. Abstention: declining to answer when uncertain.
14. Risk-Weighted Error Taxonomies: weighting errors by their real cost.

Connections

• Probability and Statistics supplies the testing and estimation used here.
• ML Engineering and MLOps turns this evidence into release gates and monitoring.