Video
Video is a time-ordered sequence of image frames:
where is the number of frames. Video modeling adds temporal structure on top of image modeling.
Common Strategies
- Frame encoder plus temporal pooling.
- 3D convolution over space and time.
- Frame or patch tokens processed by Transformers.
- Recurrent or state-space temporal models.
- Multimodal alignment with audio, subtitles, or text descriptions.
Sampling Boundary
Video models rarely consume every frame. A clip sampler defines the actual input:
where is the start time, is clip length, and is the frame stride. Event-centered sampling, random clips, sliding windows, and key-frame sampling test different abilities.
Temporal Output Spaces
Video outputs can be clip-level, frame-level, event-level, or sequence-level:
The split and metric should match the output. A clip-level label does not prove temporal localization unless the evaluation asks for event boundaries.
Leakage Risks
- Adjacent clips from the same source video appear in different splits.
- Scene, camera, uploader, or timestamp identifies the label.
- Subtitles, audio, or metadata reveal answers when the intended task is visual.
- Future frames are visible in a causal prediction task.
Checks
- What frame rate and clip length are used?
- Is sampling uniform, event-centered, or learned?
- Does the model need short motion, long-range temporal reasoning, or both?
- Are adjacent clips from the same source split across train and test?
- Are audio and text aligned to the correct temporal segment?
- Is the task allowed to use future frames?
- Are labels clip-level, frame-level, or event-boundary labels?
- Are long videos evaluated without leaking neighboring windows?