Video Understanding

Video understanding turns ordered frames into representations, events, tracks, language-facing answers, or predictions about what happens next. The section starts with representations and spatiotemporal cues, then moves through recognition architectures, temporal decision problems, multi-person systems, and world-model-style latent prediction.

Knowledge map

Representations and motion cues feed the recognition architectures; those support action recognition and streaming multi-person systems, while self-supervised pretraining leads to video-language and world models.

flowchart TD
  Rep[Video Representation and Motion] --> Arch[3D CNNs, Two-Stream, Video Transformers]
  Arch --> Recog[Action Recognition and Localization]
  Recog --> Stream[Streaming and Multi-Person Systems]
  Rep --> SSL[Self-Supervised Video and JEPA]
  SSL --> World[Video-Language and World Models]

Reading path

Read representations, then architectures, temporal and multi-person decisions, and finally foundation models and prediction.

1. Spatial and Temporal Modelling: how space and time are jointly modeled.
2. Video Representation: frames, clips, and tokens.
3. Optical Flow: dense motion between frames.
4. 3D Convolutional Networks: convolving over time as a third dimension.
5. Two-Stream Models: separate appearance and motion streams.
6. Video Transformers: attention over space-time tokens.
7. Temporal Action Recognition: classifying actions in clips.
8. Temporal Localization: finding when an action occurs.
9. Sliding Window Inference: scanning long video with windows.
10. Trigger Point Prediction: deciding the moment to act.
11. Person Tracking and Track Aggregation: linking detections into tracks.
12. Real-Time Video Understanding: latency and throughput constraints.
13. Gesture Recognition: recognizing hand and body gestures.
14. Self-Supervised Video Representation Learning: pretraining from unlabeled video.
15. Video-Language Models: connecting video to text.
16. V-JEPA: joint-embedding predictive pretraining for video.
17. V-JEPA 2: the scaled successor.
18. V-JEPA 2 versus Vision-Language Models: contrasting the two paradigms.
19. World Models: learned models of environment dynamics.
20. World Models and JEPA: predictive latent world modeling.

Connections

• Computer Vision supplies the per-frame representations extended here across time.
• Generative AI shares the foundation-model and world-model ideas.