What is agentic video understanding?

Traditional video understanding often starts by converting a video into information that a multimodal model can process. A system might sample frames at regular intervals, process the audio or transcript, and provide that information to the model as context.

This approach can work well, particularly for shorter videos. But it treats the video largely the same regardless of the question being asked.

Agentic video understanding adds another step: the model can decide how to explore the video based on the task.

A simplified process might look like this:

1. Understand the question: The model determines what information it needs from the video
2. Navigate the video: It identifies potentially relevant portions of the timeline
3. Gather information: It examines the transcript, audio, or visual frames it needs
4. Reason about what it finds: The model evaluates whether it has enough information to answer the question
5. Explore further if needed: It can inspect additional moments or gather more detail before producing a response

In other words, the model is not just reasoning about the video content. It's also reasoning about how to examine the video.

Consider a two-hour sporting event. If the question is "When did the player leave the game with an injury?", processing every frame at the same level of detail may be unnecessary. An agentic system could use available context to locate likely moments, inspect those sections more closely, and gather the visual and audio information needed to answer the question.

The main difference between static and agentic video understanding is how information from the video is selected for the model.

The ability to selectively explore a video can change both how much information a model needs to process and how it reasons about long-form content. Some of the main benefits include:

• More efficient video processing: The model can focus on relevant information instead of placing uniformly sampled content from the entire video into context.
• Better use of context: Long videos can contain far more information than is relevant to a particular question. Selective exploration leaves more room for the information that matters.
• More targeted analysis: The model can inspect specific moments more closely when a question depends on visual or temporal details.
• Flexible multimodal reasoning: Different questions may require different combinations of transcript, audio, and visual information.
• Support for longer videos: Selective processing becomes increasingly useful as video duration increases and exhaustive processing becomes more expensive.

There are tradeoffs. Dynamically exploring a video requires the model to perform additional reasoning and retrieval steps before answering. For short videos where most of the content is relevant, that extra work may provide less benefit. Applications also need to consider whether their task requires broad understanding of the entire video or targeted investigation of particular moments.

The Gemini API supports both static and agentic approaches to video processing.

With static processing, Gemini extracts video frames at a fixed rate and places them into the model context along with audio information. This can work well for many video understanding tasks, including summarization, description, question answering, and identifying information at specific timestamps.

With agentic video understanding, supported Gemini models can instead dynamically navigate the video timeline. The model can selectively inspect the transcript, frames, and audio it needs based on the prompt, and adjust how it examines the video as it works toward an answer.

For example, imagine giving Gemini a long lecture and asking, "What are the three main arguments presented?" Rather than loading uniformly sampled visual information from the entire lecture into context, the model can navigate through the video and retrieve the information it determines is relevant to the question.

This can significantly reduce the amount of video information that needs to be processed. For long-form content, Google has found that agentic video processing can use up to 88% fewer tokens than static processing while also improving response quality.

The tradeoff is that navigating the video requires additional reasoning and tool calls. For short videos, particularly those under five minutes, static processing may offer lower time to first token. Static processing can also be preferable when an application needs consistent frame-level inspection across an entire clip.

Developers can choose the processing approach based on the task, and can use different processing modes for different videos within the same request. This makes it possible to use static processing when exhaustive inspection is useful and agentic processing when selective exploration is a better fit.