Label Detection Tutorial

Audience

This tutorial is designed to help you start exploring and developing applications with the Video Intelligence API. It's designed for people with basic familiarity with Python. You should also be able to follow along with limited programming knowledge. Having walked through this tutorial, you should be able to use the Reference documentation to create your own basic applications.

This tutorial steps through a Video Intelligence API application using Python code. The purpose here is not to explain the Python client libraries, but to explain how to make calls to the Video Intelligence API using the video label detection feature. Applications in Java and Node.js are essentially similar.

If you're looking for a code-only example or an example in another language, check out the companion how-to guide.

Prerequisites

This tutorial has the following prerequisites:

You've set up a Video Intelligence API project in the Google Cloud console.
You've set up your environment using a service account and Application Default Credentials.
You have basic familiarity with Python programming.
You've set up your Python development environment.
It's recommended that you have the latest version of Python, pip, and virtualenv installed on your system. For instructions, refer to the Python Development Environment Setup Guide for Google Cloud.
You've installed the Google Cloud client library

Annotate a video using label detection

This tutorial walks you through a basic Video API application, using a LABEL_DETECTION request. A LABEL_DETECTION request annotates a video with labels (or "tags") that are selected based on the image content. For example, a video of a train at a crossing may produce labels such as "train", "transportation", "railroad crossing."

The following is the entire code needed for this tutorial. Most comments have been removed from this code to highlight how brief the code is. Instead, comments are provided later as we walk through the code.

import argparse

from google.cloud import videointelligence



def analyze_labels(path):
    """Detects labels given a GCS path."""
    video_client = videointelligence.VideoIntelligenceServiceClient()
    features = [videointelligence.Feature.LABEL_DETECTION]
    operation = video_client.annotate_video(
        request={"features": features, "input_uri": path}
    )
    print("\nProcessing video for label annotations:")

    result = operation.result(timeout=90)
    print("\nFinished processing.")

    segment_labels = result.annotation_results[0].segment_label_annotations
    for i, segment_label in enumerate(segment_labels):
        print("Video label description: {}".format(segment_label.entity.description))
        for category_entity in segment_label.category_entities:
            print(
                "\tLabel category description: {}".format(category_entity.description)
            )

        for i, segment in enumerate(segment_label.segments):
            start_time = (
                segment.segment.start_time_offset.seconds
                + segment.segment.start_time_offset.microseconds / 1e6
            )
            end_time = (
                segment.segment.end_time_offset.seconds
                + segment.segment.end_time_offset.microseconds / 1e6
            )
            positions = "{}s to {}s".format(start_time, end_time)
            confidence = segment.confidence
            print("\tSegment {}: {}".format(i, positions))
            print("\tConfidence: {}".format(confidence))
        print("\n")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("path", help="GCS file path for label detection.")
    args = parser.parse_args()

    analyze_labels(args.path)

This simple application performs the following tasks:

Imports the libraries necessary to run the application.
Takes a video file stored in Cloud Storage URI as an argument and passes it to the main() function.
Gets credentials to run the Video Intelligence API service.
Creates a video annotation request to send to the video service.
Sends the request and returns a long-running operation.
Loops over the long-running operation until the video is processed and returns available values.
Parses the response for the service and displays the response to the user.

Import libraries

import argparse

from google.cloud import videointelligence

Some standard libraries are imported: argparse to allow the application to accept input filenames as arguments and sys for formatting output while waiting for API responses. The package time is imported to run some simple wait loops.

When using the Video Intelligence API, you'll also need to import the google.cloud.videointelligence_v1 and its enumeration class, which holds the directory of our API calls.

Run the application

parser = argparse.ArgumentParser(
    description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("path", help="GCS file path for label detection.")
args = parser.parse_args()

analyze_labels(args.path)

Here, the passed argument is parsed for the Cloud Storage URI of the video filename and is passed to the main() function.

Authenticate to the API

Before communicating with the Video Intelligence API service, you need to authenticate your service using previously acquired credentials. Within an application, the simplest way to obtain credentials is to use Application Default Credentials (ADC). By default, ADC attempts to obtain credentials from the GOOGLE_APPLICATION_CREDENTIALS environment file, which should be set to point to your service account's JSON key file. (You should have set up your service account and environment to use ADC in the Quickstart.

Construct the request

video_client = videointelligence.VideoIntelligenceServiceClient()
features = [videointelligence.Feature.LABEL_DETECTION]
operation = video_client.annotate_video(
    request={"features": features, "input_uri": path}
)

Now that the Video Intelligence API service is ready, you can construct a request to that service. Requests to the Video Intelligence API are provided as JSON objects. See the Video Intelligence API Reference for complete information on the specific structure of such a request.

This code snippet performs the following tasks:

Constructs the JSON for a POST request to the annotate_video() method.
Injects the Cloud Storage location of the passed video filename into the request.
Indicates that the annotate method should perform LABEL_DETECTION.

Check the operation

result = operation.result(timeout=90)
print("\nFinished processing.")

Using the existing operation request for the existing operation, a while loop is constructed to periodically check the state of that operation. Once the operation has indicated that the operation is done, the response is parsed.

Parse the response

segment_labels = result.annotation_results[0].segment_label_annotations
for i, segment_label in enumerate(segment_labels):
    print("Video label description: {}".format(segment_label.entity.description))
    for category_entity in segment_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    for i, segment in enumerate(segment_label.segments):
        start_time = (
            segment.segment.start_time_offset.seconds
            + segment.segment.start_time_offset.microseconds / 1e6
        )
        end_time = (
            segment.segment.end_time_offset.seconds
            + segment.segment.end_time_offset.microseconds / 1e6
        )
        positions = "{}s to {}s".format(start_time, end_time)
        confidence = segment.confidence
        print("\tSegment {}: {}".format(i, positions))
        print("\tConfidence: {}".format(confidence))
    print("\n")

Once the operation has been completed, the response will contain the result within an AnnotateVideoResponse, which consists of a list of annotationResults, one for each video sent in the request. Because only one video was sent in the request, the first segmentLabelAnnotations of the results is taken, all the labels in segmentLabelAnnotations are looped through. By using only segmentLabelAnnotations, this tutorial displays only video-level annotations. Each segment_label includes a description (segment_label.description), a list of entity categories (segment_label.category_entities), and a list of segments identifying the start/end time of the label occurrences in the video (should be one segment spanning the whole video or video segment for the case of segment_label_annotations).

{
   "name":"us-west1.12089999971048628582",
   "metadata":{
      "@type":"type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
      "annotationProgress":[
         {
            "inputUri":"gs://YOUR_BUCKET/YOUR_OBJECT",
            "updateTime":"2020-01-31T01:49:52.498015Z",
            "startTime":"2020-01-31T01:49:43.056481Z"
         }
      ]
   },
   "done": true,
   "response":{
      "@type":"type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
      "annotationResults":[
         {
            "inputUri":"gs://YOUR_BUCKET/YOUR_OBJECT",
            "segmentLabelAnnotations": [
              {
                "entity": {
                  "entityId": "/m/01yrx",
                  "languageCode": "en-US"
                },
                "segments": [
                  {
                    "segment": {
                      "startTimeOffset": "0s",
                      "endTimeOffset": "14.833664s"
                    },
                    "confidence": 0.98509187
                  }
                ]
              },
               ...
            ]
         }
      ]
   }
}

Because only one video was sent in the request, the first description of the first result is printed.

Run your application

To run your application, simply pass it the Cloud Storage URI of a video:

$ python labels.py gs://YOUR_BUCKET/YOUR_OBJECT
Operation us-west1.4757250774497581229 started: 2020-01-30T01:46:30.158989Z
Operation processing ...
The video has been successfully processed.

Video label description: urban area
        Label category description: city
        Segment 0: 0.0s to 38.752016s
        Confidence: 0.946980476379


Video label description: traffic
        Segment 0: 0.0s to 38.752016s
        Confidence: 0.94105899334


Video label description: vehicle
        Segment 0: 0.0s to 38.752016s
        Confidence: 0.919958174229
...

Output

Below is an example of a possible output.

Processing video for label annotations:

Finished processing. Video label description: crowd Label category description: people Segment 0: 0.0s to 60.24s Confidence: 0.527720749378

Video label description: official Label category description: person Segment 0: 0.0s to 60.24s Confidence: 0.372822880745

Video label description: audience Label category description: people Segment 0: 0.0s to 60.24s Confidence: 0.501719772816

Video label description: news Segment 0: 0.0s to 60.24s Confidence: 0.867252230644

Video label description: people Label category description: person Segment 0: 0.0s to 60.24s Confidence: 0.46747264266

Video label description: politics Segment 0: 0.0s to 60.24s Confidence: 0.319397002459

Congratulations! You've performed an annotation task using the Video Intelligence API!