3D Convolutional Networks

3D convolutional networks replace image kernels with kernels over $(T,H,W)$. A 2D CNN sees one frame at a time; a 3D CNN can fire on a short motion pattern such as “hand moves upward while the torso stays still.” They are a direct architecture for spatial and temporal modelling, an alternative to two-stream models, and a useful baseline before reaching for video transformers.

The 3D convolution

For input $X\in {\mathbb{R}}^{C_{in}\times T\times H\times W}$, a 3D convolution computes

$$Y_{c_o,t,i,j}=b_{c_o}+\sum _{c_i}\sum _{\tau ,u,v}{W}_{c_o,c_i,\tau ,u,v}{X}_{c_i,t+\tau ,i+u,j+v}.$$

The temporal kernel size controls the motion horizon visible to one layer. Stacking layers grows the temporal receptive field, while strides trade temporal resolution for compute. This is why 3D CNN features are often pooled into a compact video representation for downstream recognition or retrieval.

Worked example

This snippet applies a 3D convolution over time, height, and width and checks that the output shape follows the manual convolution-size formula.

import torch
 
torch.manual_seed(4)
conv = torch.nn.Conv3d(1, 2, kernel_size=(3,3,3), stride=(2,1,1), padding=(1,1,1), bias=False)
x = torch.arange(1*1*8*6*6, dtype=torch.float32).reshape(1,1,8,6,6) / 100
with torch.no_grad():
    conv.weight.fill_(1/27)
y = conv(x)
manual = [(8+2*1-3)//2+1, (6+2*1-3)//1+1, (6+2*1-3)//1+1]
print("output_shape", list(y.shape), "manual_DHW", manual)
print("first_channel_t0_patch", torch.round(y[0,0,0,:2,:3], decimals=3).tolist())

Observed output:

output_shape [1, 2, 4, 6, 6] manual_DHW [4, 6, 6]
first_channel_t0_patch [[0.06400000303983688, 0.09799999743700027, 0.10199999809265137], [0.10899999737739563, 0.16699999570846558, 0.17299999296665192]]

The temporal stride halves the eight-frame clip to four temporal positions. The kernel is an average here, but learned kernels specialize into appearance-motion patterns.

The snippet is an API example as much as a calculation: PyTorch Conv3d expects input shaped as batch, channels, time/depth, height, width. The manual depth-height-width calculation uses the standard convolution-size formula, so the output shape [1, 2, 4, 6, 6] means two learned filters were applied at four temporal positions over the original spatial grid. The first patch values are smaller near the boundary because padding inserts zeros around the clip.

Caveats

3D kernels are parameter- and memory-heavy because activations retain time as well as space. Short clips can miss long-range context, while aggressive temporal stride hurts temporal localization. Pretraining and careful sampling matter because video labels are expensive and adjacent frames are highly redundant.

References

• Tran et al., 2015, Learning Spatiotemporal Features with 3D Convolutional Networks
• Carreira and Zisserman, 2017, Quo Vadis, Action Recognition?
• PyTorch documentation: Conv3d

Nav

Section — Video Understanding

← Optical Flow Two-Stream Models →