Forecast Calibration

Forecast calibration checks whether predicted probabilities match observed frequencies. In forecasting, calibration is usually assessed for quantiles, prediction intervals, or full predictive distributions.

If a model issues many 80 percent prediction intervals, about 80 percent of the corresponding future observations should fall inside those intervals for comparable cases.

Quantile calibration

For a quantile forecast ${\hat{q}}_{\tau ,i}$, empirical quantile coverage is:

$${\hat{c}}_{\tau }=\frac{1}{n}\sum _{i=1}^{n}1\{y_i\le {\hat{q}}_{\tau ,i}\}$$

A calibrated $\tau$-quantile has ${\hat{c}}_{\tau }\approx \tau$. For example, if a 0.9 quantile is exceeded 25 percent of the time, it is too low.

Interval calibration

For intervals $[L_i,U_i]$, empirical coverage measures how often observations fall inside the interval. Coverage should be reported with interval width. Wide intervals can be calibrated but uninformative.

Calibration should be checked by horizon, entity group, season, promotion state, and volume level. Aggregate calibration can hide under-coverage in the exact segments where uncertainty matters most.

Recalibration

Recalibration adjusts predicted quantiles or intervals using validation data. Examples include multiplying interval widths by a horizon-specific factor, mapping nominal quantile levels to empirical levels, or applying conformal prediction.

Recalibration data must be separate from the final evaluation period. Otherwise the final coverage estimate is optimistic.

Practical guidance

• Plot empirical coverage against nominal quantile or interval levels.
• Report calibration by horizon and important business segments.
• Track both coverage and sharpness.
• Recalibrate only on validation data that would have been available before final evaluation.
• Recheck calibration after major feature, model, or data pipeline changes.

Common failure modes

• Reporting average coverage while ignoring long-horizon under-coverage.
• Making intervals wider until coverage looks good without tracking usefulness.
• Treating attention weights or feature importance as evidence of calibration.
• Recalibrating on the same period used for final reporting.
• Ignoring delayed labels when monitoring live calibration.

Connections

Calibration is the reliability check for probabilistic forecasting. It complements prediction intervals, quantile loss, conformal prediction for forecasting, and the broader evaluation notion of calibration.

References

• Hyndman & Athanasopoulos, FPP3: Distributional forecasts and prediction intervals
• Romano, Patterson, and Candes, Conformalized Quantile Regression

Nav

Section — Time-Series Forecasting

← Quantile Loss Conformal Prediction for Forecasting →