Transformer-Based Forecasting

Transformer forecasters adapt attention to temporal data. Instead of carrying a recurrent state, the model forms query, key, and value representations for time steps or patches and combines history through attention weights:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V.$$

For forecasting, token design matters as much as the attention equation. Tokens may represent individual time steps, patches of a long sequence, variables, or entity-time combinations. Inputs often combine past targets, observed covariates, known future covariates, static entity features, and positional or calendar encodings. The forecast head then emits one or more horizons, sometimes as quantiles for probabilistic forecasting.

Temporal Fusion Transformer is a well-known example for multi-horizon forecasting. It combines recurrent local processing, interpretable variable selection, gating, static covariate encoders, and attention over temporal features. Other transformer variants reduce attention cost or patch long histories to make longer context practical.

The main failure mode is assuming attention solves time-series structure by default. A transformer can attend to leaked future covariates, memorize entity identity, or waste capacity on short series where exponential smoothing, ARIMA, or gradient-boosted lag features are stronger. Backtests should be segmented by horizon, series age, volatility, and calendar regime before attributing gains to long-range context.

The architecture is most defensible when there are many related series, important known future covariates, or interactions across variables that simpler lag models miss. For a small univariate series, attention often adds variance before it adds signal.

Connections

Transformer forecasting adapts transformers to temporal data. It is one branch of deep learning forecasting, alongside RNN and LSTM forecasting, temporal convolutional networks, and N-BEATS and N-HiTS.

References

• Lim et al., Temporal Fusion Transformers
• Vaswani et al., Attention Is All You Need
• Nixtla NeuralForecast documentation

Nav

Section — Time-Series Forecasting

← Temporal Convolutional Networks N-BEATS and N-HiTS →