Analyzing videos for labels

The Video Intelligence API can identify entities shown in video footage using the LABEL_DETECTION feature. This feature identifies objects, locations, activities, animal species, products, and more.

The analysis can be compartmentalized as follows:

• Frame level:
  Entities are identified and labeled within each frame (with one frame per second sampling).
• Shot level:
  Shots are automatically detected within every segment (or video). Entities are then identified and labeled within each shot.
• Segment level:
  User-selected segments of a video can be specified for analysis by stipulating beginning and ending timestamps for the purposes of annotation (see VideoSegment). Entities are then identified and labeled within each segment. If no segments are specified, the whole video is treated as one segment.

Note: For an example, see the Label detection tutorial

Annotating a local file

Here is an example of performing video analysis for labels on a local file.

Looking for something more in-depth? Check out our detailed Python tutorial.

REST & CMD LINE

Send the process request

The following shows how to send a POST request to the videos:annotate method. You can configure the LabelDetectionMode to shot-level and/or frame-level annotations. We recommend using SHOT_AND_FRAME_MODE. The example uses the access token for a service account set up for the project using the Cloud SDK. For instructions on installing the Cloud SDK, setting up a project with a service account, and obtaining an access token, see the Video Intelligence quickstart.

Before using any of the request data below, make the following replacements:

• base64-encoded-content: your video as base64 encoded data. See the instructions on how to convert your data to base64.

HTTP method and URL:

POST https://videointelligence.googleapis.com/v1/videos:annotate

Request JSON body:

{
  "inputContent": "base64-encoded-content",
  "features": ["LABEL_DETECTION"],
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Save the request body in a file called request.json, and execute the following command:

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
https://videointelligence.googleapis.com/v1/videos:annotate

PowerShell (Windows)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Save the request body in a file called request.json, and execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://videointelligence.googleapis.com/v1/videos:annotate" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/project-number/locations/location-id/operations/operation-id"
}

If the request is successful, Video Intelligence returns the name of your operation.

Get the results

To get the results of your request, you must send a GET request to the projects.locations.operations resource. The following shows how to send such a request.

Before using any of the request data below, make the following replacements:

• operation-name: the name of the operation as returned by the Video Intelligence API. The operation name has the format projects/project-number/locations/location-id/operations/operation-id

HTTP method and URL:

GET https://videointelligence.googleapis.com/v1/operation-name

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Execute the following command:

curl -X GET \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
https://videointelligence.googleapis.com/v1/operation-name

PowerShell (Windows)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://videointelligence.googleapis.com/v1/operation-name" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{
  "name": "projects/project-number/locations/location-id/operations/operation-id",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "progressPercent": 100,
        "startTime": "2019-03-12T19:36:09.110351Z",
        "updateTime": "2019-03-12T19:36:17.519069Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "segmentLabelAnnotations": [
          {
            "entity": {
              "entityId": "/m/01prls",
              "description": "land vehicle",
              "languageCode": "en-US"
            },
            "categoryEntities": [
              {
                "entityId": "/m/07yv9",
                "description": "vehicle",
                "languageCode": "en-US"
              }
            ],
            "segments": [
              {
                "segment": {
                  "startTimeOffset": "0s",
                  "endTimeOffset": "38.757872s"
                },
                "confidence": 0.6614419
              }
            ]
          },
          {
            "entity": {
              "entityId": "/m/039jbq",
              "description": "urban area",
              "languageCode": "en-US"
            },
            "categoryEntities": [
              {
                "entityId": "/m/01n32",
                "description": "city",
                "languageCode": "en-US"
              }
            ],
            "segments": [
              {
                "segment": {
                  "startTimeOffset": "0s",
                  "endTimeOffset": "38.757872s"
                },
                "confidence": 0.92337775
              }
            ]
          },
          ...

C#

View on GitHub Feedback

public static object AnalyzeLabels(string path)
{
    var client = VideoIntelligenceServiceClient.Create();
    var request = new AnnotateVideoRequest()
    {
        InputContent = Google.Protobuf.ByteString.CopyFrom(File.ReadAllBytes(path)),
        Features = { Feature.LabelDetection },
    };
    var op = client.AnnotateVideo(request).PollUntilCompleted();
    foreach (var result in op.Result.AnnotationResults)
    {
        PrintLabels("Video", result.SegmentLabelAnnotations);
        PrintLabels("Shot", result.ShotLabelAnnotations);
        PrintLabels("Frame", result.FrameLabelAnnotations);
    }
    return 0;
}

static void PrintLabels(string labelName,
    IEnumerable<LabelAnnotation> labelAnnotations)
{
    foreach (var annotation in labelAnnotations)
    {
        Console.WriteLine($"{labelName} label: {annotation.Entity.Description}");
        foreach (var entity in annotation.CategoryEntities)
        {
            Console.WriteLine($"{labelName} label category: {entity.Description}");
        }
        foreach (var segment in annotation.Segments)
        {
            Console.Write("Segment location: ");
            Console.Write(segment.Segment.StartTimeOffset);
            Console.Write(":");
            Console.WriteLine(segment.Segment.EndTimeOffset);
            System.Console.WriteLine($"Confidence: {segment.Confidence}");
        }
    }
}

Go

View on GitHub Feedback

func label(w io.Writer, file string) error {
	ctx := context.Background()
	client, err := video.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("video.NewClient: %v", err)
	}

	fileBytes, err := ioutil.ReadFile(file)
	if err != nil {
		return err
	}

	op, err := client.AnnotateVideo(ctx, &videopb.AnnotateVideoRequest{
		Features: []videopb.Feature{
			videopb.Feature_LABEL_DETECTION,
		},
		InputContent: fileBytes,
	})
	if err != nil {
		return fmt.Errorf("AnnotateVideo: %v", err)
	}

	resp, err := op.Wait(ctx)
	if err != nil {
		return fmt.Errorf("Wait: %v", err)
	}

	printLabels := func(labels []*videopb.LabelAnnotation) {
		for _, label := range labels {
			fmt.Fprintf(w, "\tDescription: %s\n", label.Entity.Description)
			for _, category := range label.CategoryEntities {
				fmt.Fprintf(w, "\t\tCategory: %s\n", category.Description)
			}
			for _, segment := range label.Segments {
				start, _ := ptypes.Duration(segment.Segment.StartTimeOffset)
				end, _ := ptypes.Duration(segment.Segment.EndTimeOffset)
				fmt.Fprintf(w, "\t\tSegment: %s to %s\n", start, end)
			}
		}
	}

	// A single video was processed. Get the first result.
	result := resp.AnnotationResults[0]

	fmt.Fprintln(w, "SegmentLabelAnnotations:")
	printLabels(result.SegmentLabelAnnotations)
	fmt.Fprintln(w, "ShotLabelAnnotations:")
	printLabels(result.ShotLabelAnnotations)
	fmt.Fprintln(w, "FrameLabelAnnotations:")
	printLabels(result.FrameLabelAnnotations)

	return nil
}

Java

View on GitHub Feedback

// Instantiate a com.google.cloud.videointelligence.v1.VideoIntelligenceServiceClient
try (VideoIntelligenceServiceClient client = VideoIntelligenceServiceClient.create()) {
  // Read file and encode into Base64
  Path path = Paths.get(filePath);
  byte[] data = Files.readAllBytes(path);

  AnnotateVideoRequest request =
      AnnotateVideoRequest.newBuilder()
          .setInputContent(ByteString.copyFrom(data))
          .addFeatures(Feature.LABEL_DETECTION)
          .build();
  // Create an operation that will contain the response when the operation completes.
  OperationFuture<AnnotateVideoResponse, AnnotateVideoProgress> response =
      client.annotateVideoAsync(request);

  System.out.println("Waiting for operation to complete...");
  for (VideoAnnotationResults results : response.get().getAnnotationResultsList()) {
    // process video / segment level label annotations
    System.out.println("Locations: ");
    for (LabelAnnotation labelAnnotation : results.getSegmentLabelAnnotationsList()) {
      System.out.println("Video label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Video label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.2f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }

    // process shot label annotations
    for (LabelAnnotation labelAnnotation : results.getShotLabelAnnotationsList()) {
      System.out.println("Shot label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Shot label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.2f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }

    // process frame label annotations
    for (LabelAnnotation labelAnnotation : results.getFrameLabelAnnotationsList()) {
      System.out.println("Frame label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Frame label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.2f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }
  }
}

Node.js

View on GitHub Feedback

// Imports the Google Cloud Video Intelligence library + Node's fs library
const video = require('@google-cloud/video-intelligence').v1;
const fs = require('fs');
const util = require('util');

// Creates a client
const client = new video.VideoIntelligenceServiceClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const path = 'Local file to analyze, e.g. ./my-file.mp4';

// Reads a local video file and converts it to base64
const readFile = util.promisify(fs.readFile);
const file = await readFile(path);
const inputContent = file.toString('base64');

// Constructs request
const request = {
  inputContent: inputContent,
  features: ['LABEL_DETECTION'],
};

// Detects labels in a video
const [operation] = await client.annotateVideo(request);
console.log('Waiting for operation to complete...');
const [operationResult] = await operation.promise();
// Gets annotations for video
const annotations = operationResult.annotationResults[0];

const labels = annotations.segmentLabelAnnotations;
labels.forEach(label => {
  console.log(`Label ${label.entity.description} occurs at:`);
  label.segments.forEach(segment => {
    const time = segment.segment;
    if (time.startTimeOffset.seconds === undefined) {
      time.startTimeOffset.seconds = 0;
    }
    if (time.startTimeOffset.nanos === undefined) {
      time.startTimeOffset.nanos = 0;
    }
    if (time.endTimeOffset.seconds === undefined) {
      time.endTimeOffset.seconds = 0;
    }
    if (time.endTimeOffset.nanos === undefined) {
      time.endTimeOffset.nanos = 0;
    }
    console.log(
      `\tStart: ${time.startTimeOffset.seconds}` +
        `.${(time.startTimeOffset.nanos / 1e6).toFixed(0)}s`
    );
    console.log(
      `\tEnd: ${time.endTimeOffset.seconds}.` +
        `${(time.endTimeOffset.nanos / 1e6).toFixed(0)}s`
    );
    console.log(`\tConfidence: ${segment.confidence}`);
  });
});

Python

For more information on installing and using the Cloud Video Intelligence API Client Library for Python, refer to Cloud Video Intelligence API Client Libraries.

View on GitHub Feedback

"""Detect labels given a file path."""
video_client = videointelligence.VideoIntelligenceServiceClient()
features = [videointelligence.Feature.LABEL_DETECTION]

with io.open(path, "rb") as movie:
    input_content = movie.read()

operation = video_client.annotate_video(
    request={"features": features, "input_content": input_content}
)
print("\nProcessing video for label annotations:")

result = operation.result(timeout=90)
print("\nFinished processing.")

# Process video/segment level label annotations
segment_labels = result.annotation_results[0].segment_label_annotations
for i, segment_label in enumerate(segment_labels):
    print("Video label description: {}".format(segment_label.entity.description))
    for category_entity in segment_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    for i, segment in enumerate(segment_label.segments):
        start_time = (
            segment.segment.start_time_offset.seconds
            + segment.segment.start_time_offset.microseconds / 1e6
        )
        end_time = (
            segment.segment.end_time_offset.seconds
            + segment.segment.end_time_offset.microseconds / 1e6
        )
        positions = "{}s to {}s".format(start_time, end_time)
        confidence = segment.confidence
        print("\tSegment {}: {}".format(i, positions))
        print("\tConfidence: {}".format(confidence))
    print("\n")

# Process shot level label annotations
shot_labels = result.annotation_results[0].shot_label_annotations
for i, shot_label in enumerate(shot_labels):
    print("Shot label description: {}".format(shot_label.entity.description))
    for category_entity in shot_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    for i, shot in enumerate(shot_label.segments):
        start_time = (
            shot.segment.start_time_offset.seconds
            + shot.segment.start_time_offset.microseconds / 1e6
        )
        end_time = (
            shot.segment.end_time_offset.seconds
            + shot.segment.end_time_offset.microseconds / 1e6
        )
        positions = "{}s to {}s".format(start_time, end_time)
        confidence = shot.confidence
        print("\tSegment {}: {}".format(i, positions))
        print("\tConfidence: {}".format(confidence))
    print("\n")

# Process frame level label annotations
frame_labels = result.annotation_results[0].frame_label_annotations
for i, frame_label in enumerate(frame_labels):
    print("Frame label description: {}".format(frame_label.entity.description))
    for category_entity in frame_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    # Each frame_label_annotation has many frames,
    # here we print information only about the first frame.
    frame = frame_label.frames[0]
    time_offset = frame.time_offset.seconds + frame.time_offset.microseconds / 1e6
    print("\tFirst frame time offset: {}s".format(time_offset))
    print("\tFirst frame confidence: {}".format(frame.confidence))
    print("\n")

PHP

View on GitHub Feedback

use Google\Cloud\VideoIntelligence\V1\VideoIntelligenceServiceClient;
use Google\Cloud\VideoIntelligence\V1\Feature;

/** Uncomment and populate these variables in your code */
// $path = 'File path to a video file to analyze';
// $options = [];

# Instantiate a client.
$video = new VideoIntelligenceServiceClient();

# Read the local video file
$inputContent = file_get_contents($path);

# Execute a request.
$operation = $video->annotateVideo([
    'inputContent' => $inputContent,
    'features' => [Feature::LABEL_DETECTION]
]);

# Wait for the request to complete.
$operation->pollUntilComplete($options);

# Print the results.
if ($operation->operationSucceeded()) {
    $results = $operation->getResult()->getAnnotationResults()[0];

    # Process video/segment level label annotations
    foreach ($results->getSegmentLabelAnnotations() as $label) {
        printf('Video label description: %s' . PHP_EOL, $label->getEntity()->getDescription());
        foreach ($label->getCategoryEntities() as $categoryEntity) {
            printf('  Category: %s' . PHP_EOL, $categoryEntity->getDescription());
        }
        foreach ($label->getSegments() as $segment) {
            $start = $segment->getSegment()->getStartTimeOffset();
            $end = $segment->getSegment()->getEndTimeOffset();
            printf('  Segment: %ss to %ss' . PHP_EOL,
                $start->getSeconds() + $start->getNanos()/1000000000.0,
                $end->getSeconds() + $end->getNanos()/1000000000.0);
            printf('  Confidence: %f' . PHP_EOL, $segment->getConfidence());
        }
    }
    print(PHP_EOL);

    # Process shot level label annotations
    foreach ($results->getShotLabelAnnotations() as $label) {
        printf('Shot label description: %s' . PHP_EOL, $label->getEntity()->getDescription());
        foreach ($label->getCategoryEntities() as $categoryEntity) {
            printf('  Category: %s' . PHP_EOL, $categoryEntity->getDescription());
        }
        foreach ($label->getSegments() as $shot) {
            $start = $shot->getSegment()->getStartTimeOffset();
            $end = $shot->getSegment()->getEndTimeOffset();
            printf('  Shot: %ss to %ss' . PHP_EOL,
                $start->getSeconds() + $start->getNanos()/1000000000.0,
                $end->getSeconds() + $end->getNanos()/1000000000.0);
            printf('  Confidence: %f' . PHP_EOL, $shot->getConfidence());
        }
    }
    print(PHP_EOL);
} else {
    print_r($operation->getError());
}

Ruby

View on GitHub Feedback

# path = "Path to a local video file: path/to/file.mp4"

require "google/cloud/video_intelligence"

video = Google::Cloud::VideoIntelligence.video_intelligence_service

video_contents = File.binread path

# Register a callback during the method call
operation = video.annotate_video features: [:LABEL_DETECTION], input_content: video_contents

puts "Processing video for label annotations:"
operation.wait_until_done!

raise operation.results.message? if operation.error?
puts "Finished Processing."

labels = operation.results.annotation_results.first.segment_label_annotations
print_labels labels

Annotating a file on Cloud Storage

Here is an example of performing video analysis for labels on a file located in Cloud Storage.

REST & CMD LINE

Send the process request

The following shows how to send a POST request to the annotate method. The example uses the access token for a service account set up for the project using the Cloud SDK. For instructions on installing the Cloud SDK, setting up a project with a service account, and obtaining an access token, see the Video Intelligence quickstart.

Before using any of the request data below, make the following replacements:

• input-uri: a Cloud Storage bucket that contains the file you want to annotate, including the file name. Must start with gs://.

HTTP method and URL:

POST https://videointelligence.googleapis.com/v1/videos:annotate

Request JSON body:

{
  "inputUri": "input-uri",
  "features": ["LABEL_DETECTION"],
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Save the request body in a file called request.json, and execute the following command:

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
https://videointelligence.googleapis.com/v1/videos:annotate

PowerShell (Windows)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Save the request body in a file called request.json, and execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://videointelligence.googleapis.com/v1/videos:annotate" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/project-number/locations/location-id/operations/operation-id"
}

If the request is successful, the Video Intelligence returns the name of your operation.

Get the results

To get the results of your request, you must send a GET request to the projects.locations.operations resource. The following shows how to send such a request.

Before using any of the request data below, make the following replacements:

• operation-name: the name of the operation as returned by the Video Intelligence API. The operation name has the format projects/project-number/locations/location-id/operations/operation-id

HTTP method and URL:

GET https://videointelligence.googleapis.com/v1/operation-name

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Execute the following command:

curl -X GET \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
https://videointelligence.googleapis.com/v1/operation-name

PowerShell (Windows)

Note: Ensure you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account private key file path.

Execute the following command:

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://videointelligence.googleapis.com/v1/operation-name" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

Response

{
  "name": "projects/project-number/locations/location-id/operations/operation-id",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoProgress",
    "annotationProgress": [
      {
        "inputUri": "input-uri",
        "progressPercent": 100,
        "startTime": "2019-11-13T19:25:56.206335Z",
        "updateTime": "2019-11-13T19:26:16.215615Z"
      }
    ]
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.videointelligence.v1.AnnotateVideoResponse",
    "annotationResults": [
      {
        "inputUri": "input-uri",
        "segmentLabelAnnotations": [
          {
            "entity": {
              "entityId": "/m/01vkl",
              "description": "circle",
              "languageCode": "en-US"
            },
            "categoryEntities": [
              {
                "entityId": "/m/016nqd",
                "description": "shape",
                "languageCode": "en-US"
              }
            ],
            "segments": [
              {
                "segment": {
                  "startTimeOffset": "0s",
                  "endTimeOffset": "16.416666s"
                },
                "confidence": 0.36535457
              }
            ]
          },
          ...
        ]
     }
    ]
  }
}

Download annotation results

Copy the annotation from the source to the destination bucket: (see Copy files and objects)

gsutil cp gcs_uri gs://my-bucket

Note: If the output gcs uri is provided by the user, then the annotation is stored in that gcs uri.

C#

View on GitHub Feedback

public static object AnalyzeLabelsGcs(string uri)
{
    var client = VideoIntelligenceServiceClient.Create();
    var request = new AnnotateVideoRequest()
    {
        InputUri = uri,
        Features = { Feature.LabelDetection }
    };
    var op = client.AnnotateVideo(request).PollUntilCompleted();
    foreach (var result in op.Result.AnnotationResults)
    {
        PrintLabels("Video", result.SegmentLabelAnnotations);
        PrintLabels("Shot", result.ShotLabelAnnotations);
        PrintLabels("Frame", result.FrameLabelAnnotations);
    }
    return 0;
}

static void PrintLabels(string labelName,
    IEnumerable<LabelAnnotation> labelAnnotations)
{
    foreach (var annotation in labelAnnotations)
    {
        Console.WriteLine($"{labelName} label: {annotation.Entity.Description}");
        foreach (var entity in annotation.CategoryEntities)
        {
            Console.WriteLine($"{labelName} label category: {entity.Description}");
        }
        foreach (var segment in annotation.Segments)
        {
            Console.Write("Segment location: ");
            Console.Write(segment.Segment.StartTimeOffset);
            Console.Write(":");
            Console.WriteLine(segment.Segment.EndTimeOffset);
            System.Console.WriteLine($"Confidence: {segment.Confidence}");
        }
    }
}

Go

View on GitHub Feedback

func labelURI(w io.Writer, file string) error {
	ctx := context.Background()
	client, err := video.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("video.NewClient: %v", err)
	}

	op, err := client.AnnotateVideo(ctx, &videopb.AnnotateVideoRequest{
		Features: []videopb.Feature{
			videopb.Feature_LABEL_DETECTION,
		},
		InputUri: file,
	})
	if err != nil {
		return fmt.Errorf("AnnotateVideo: %v", err)
	}

	resp, err := op.Wait(ctx)
	if err != nil {
		return fmt.Errorf("Wait: %v", err)
	}

	printLabels := func(labels []*videopb.LabelAnnotation) {
		for _, label := range labels {
			fmt.Fprintf(w, "\tDescription: %s\n", label.Entity.Description)
			for _, category := range label.CategoryEntities {
				fmt.Fprintf(w, "\t\tCategory: %s\n", category.Description)
			}
			for _, segment := range label.Segments {
				start, _ := ptypes.Duration(segment.Segment.StartTimeOffset)
				end, _ := ptypes.Duration(segment.Segment.EndTimeOffset)
				fmt.Fprintf(w, "\t\tSegment: %s to %s\n", start, end)
			}
		}
	}

	// A single video was processed. Get the first result.
	result := resp.AnnotationResults[0]

	fmt.Fprintln(w, "SegmentLabelAnnotations:")
	printLabels(result.SegmentLabelAnnotations)
	fmt.Fprintln(w, "ShotLabelAnnotations:")
	printLabels(result.ShotLabelAnnotations)
	fmt.Fprintln(w, "FrameLabelAnnotations:")
	printLabels(result.FrameLabelAnnotations)

	return nil
}

Java

View on GitHub Feedback

// Instantiate a com.google.cloud.videointelligence.v1.VideoIntelligenceServiceClient
try (VideoIntelligenceServiceClient client = VideoIntelligenceServiceClient.create()) {
  // Provide path to file hosted on GCS as "gs://bucket-name/..."
  AnnotateVideoRequest request =
      AnnotateVideoRequest.newBuilder()
          .setInputUri(gcsUri)
          .addFeatures(Feature.LABEL_DETECTION)
          .build();
  // Create an operation that will contain the response when the operation completes.
  OperationFuture<AnnotateVideoResponse, AnnotateVideoProgress> response =
      client.annotateVideoAsync(request);

  System.out.println("Waiting for operation to complete...");
  for (VideoAnnotationResults results : response.get().getAnnotationResultsList()) {
    // process video / segment level label annotations
    System.out.println("Locations: ");
    for (LabelAnnotation labelAnnotation : results.getSegmentLabelAnnotationsList()) {
      System.out.println("Video label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Video label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.3f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }

    // process shot label annotations
    for (LabelAnnotation labelAnnotation : results.getShotLabelAnnotationsList()) {
      System.out.println("Shot label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Shot label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.3f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }

    // process frame label annotations
    for (LabelAnnotation labelAnnotation : results.getFrameLabelAnnotationsList()) {
      System.out.println("Frame label: " + labelAnnotation.getEntity().getDescription());
      // categories
      for (Entity categoryEntity : labelAnnotation.getCategoryEntitiesList()) {
        System.out.println("Frame label category: " + categoryEntity.getDescription());
      }
      // segments
      for (LabelSegment segment : labelAnnotation.getSegmentsList()) {
        double startTime =
            segment.getSegment().getStartTimeOffset().getSeconds()
                + segment.getSegment().getStartTimeOffset().getNanos() / 1e9;
        double endTime =
            segment.getSegment().getEndTimeOffset().getSeconds()
                + segment.getSegment().getEndTimeOffset().getNanos() / 1e9;
        System.out.printf("Segment location: %.3f:%.2f\n", startTime, endTime);
        System.out.println("Confidence: " + segment.getConfidence());
      }
    }
  }
}

Node.js

View on GitHub Feedback

// Imports the Google Cloud Video Intelligence library
const video = require('@google-cloud/video-intelligence').v1;

// Creates a client
const client = new video.VideoIntelligenceServiceClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const gcsUri = 'GCS URI of the video to analyze, e.g. gs://my-bucket/my-video.mp4';

const request = {
  inputUri: gcsUri,
  features: ['LABEL_DETECTION'],
};

// Detects labels in a video
const [operation] = await client.annotateVideo(request);
console.log('Waiting for operation to complete...');
const [operationResult] = await operation.promise();

// Gets annotations for video
const annotations = operationResult.annotationResults[0];

const labels = annotations.segmentLabelAnnotations;
labels.forEach(label => {
  console.log(`Label ${label.entity.description} occurs at:`);
  label.segments.forEach(segment => {
    const time = segment.segment;
    if (time.startTimeOffset.seconds === undefined) {
      time.startTimeOffset.seconds = 0;
    }
    if (time.startTimeOffset.nanos === undefined) {
      time.startTimeOffset.nanos = 0;
    }
    if (time.endTimeOffset.seconds === undefined) {
      time.endTimeOffset.seconds = 0;
    }
    if (time.endTimeOffset.nanos === undefined) {
      time.endTimeOffset.nanos = 0;
    }
    console.log(
      `\tStart: ${time.startTimeOffset.seconds}` +
        `.${(time.startTimeOffset.nanos / 1e6).toFixed(0)}s`
    );
    console.log(
      `\tEnd: ${time.endTimeOffset.seconds}.` +
        `${(time.endTimeOffset.nanos / 1e6).toFixed(0)}s`
    );
    console.log(`\tConfidence: ${segment.confidence}`);
  });
});

Python

View on GitHub Feedback

""" Detects labels given a GCS path. """
video_client = videointelligence.VideoIntelligenceServiceClient()
features = [videointelligence.Feature.LABEL_DETECTION]

mode = videointelligence.LabelDetectionMode.SHOT_AND_FRAME_MODE
config = videointelligence.LabelDetectionConfig(label_detection_mode=mode)
context = videointelligence.VideoContext(label_detection_config=config)

operation = video_client.annotate_video(
    request={"features": features, "input_uri": path, "video_context": context}
)
print("\nProcessing video for label annotations:")

result = operation.result(timeout=180)
print("\nFinished processing.")

# Process video/segment level label annotations
segment_labels = result.annotation_results[0].segment_label_annotations
for i, segment_label in enumerate(segment_labels):
    print("Video label description: {}".format(segment_label.entity.description))
    for category_entity in segment_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    for i, segment in enumerate(segment_label.segments):
        start_time = (
            segment.segment.start_time_offset.seconds
            + segment.segment.start_time_offset.microseconds / 1e6
        )
        end_time = (
            segment.segment.end_time_offset.seconds
            + segment.segment.end_time_offset.microseconds / 1e6
        )
        positions = "{}s to {}s".format(start_time, end_time)
        confidence = segment.confidence
        print("\tSegment {}: {}".format(i, positions))
        print("\tConfidence: {}".format(confidence))
    print("\n")

# Process shot level label annotations
shot_labels = result.annotation_results[0].shot_label_annotations
for i, shot_label in enumerate(shot_labels):
    print("Shot label description: {}".format(shot_label.entity.description))
    for category_entity in shot_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    for i, shot in enumerate(shot_label.segments):
        start_time = (
            shot.segment.start_time_offset.seconds
            + shot.segment.start_time_offset.microseconds / 1e6
        )
        end_time = (
            shot.segment.end_time_offset.seconds
            + shot.segment.end_time_offset.microseconds / 1e6
        )
        positions = "{}s to {}s".format(start_time, end_time)
        confidence = shot.confidence
        print("\tSegment {}: {}".format(i, positions))
        print("\tConfidence: {}".format(confidence))
    print("\n")

# Process frame level label annotations
frame_labels = result.annotation_results[0].frame_label_annotations
for i, frame_label in enumerate(frame_labels):
    print("Frame label description: {}".format(frame_label.entity.description))
    for category_entity in frame_label.category_entities:
        print(
            "\tLabel category description: {}".format(category_entity.description)
        )

    # Each frame_label_annotation has many frames,
    # here we print information only about the first frame.
    frame = frame_label.frames[0]
    time_offset = frame.time_offset.seconds + frame.time_offset.microseconds / 1e6
    print("\tFirst frame time offset: {}s".format(time_offset))
    print("\tFirst frame confidence: {}".format(frame.confidence))
    print("\n")

PHP

View on GitHub Feedback

use Google\Cloud\VideoIntelligence\V1\VideoIntelligenceServiceClient;
use Google\Cloud\VideoIntelligence\V1\Feature;

/** Uncomment and populate these variables in your code */
// $uri = 'The cloud storage object to analyze (gs://your-bucket-name/your-object-name)';
// $options = [];

# Instantiate a client.
$video = new VideoIntelligenceServiceClient();

# Execute a request.
$operation = $video->annotateVideo([
    'inputUri' => $uri,
    'features' => [Feature::LABEL_DETECTION]
]);

# Wait for the request to complete.
$operation->pollUntilComplete($options);

# Print the results.
if ($operation->operationSucceeded()) {
    $results = $operation->getResult()->getAnnotationResults()[0];

    # Process video/segment level label annotations
    foreach ($results->getSegmentLabelAnnotations() as $label) {
        printf('Video label description: %s' . PHP_EOL, $label->getEntity()->getDescription());
        foreach ($label->getCategoryEntities() as $categoryEntity) {
            printf('  Category: %s' . PHP_EOL, $categoryEntity->getDescription());
        }
        foreach ($label->getSegments() as $segment) {
            $start = $segment->getSegment()->getStartTimeOffset();
            $end = $segment->getSegment()->getEndTimeOffset();
            printf('  Segment: %ss to %ss' . PHP_EOL,
                $start->getSeconds() + $start->getNanos()/1000000000.0,
                $end->getSeconds() + $end->getNanos()/1000000000.0);
            printf('  Confidence: %f' . PHP_EOL, $segment->getConfidence());
        }
    }
    print(PHP_EOL);

    # Process shot level label annotations
    foreach ($results->getShotLabelAnnotations() as $label) {
        printf('Shot label description: %s' . PHP_EOL, $label->getEntity()->getDescription());
        foreach ($label->getCategoryEntities() as $categoryEntity) {
            printf('  Category: %s' . PHP_EOL, $categoryEntity->getDescription());
        }
        foreach ($label->getSegments() as $shot) {
            $start = $shot->getSegment()->getStartTimeOffset();
            $end = $shot->getSegment()->getEndTimeOffset();
            printf('  Shot: %ss to %ss' . PHP_EOL,
                $start->getSeconds() + $start->getNanos()/1000000000.0,
                $end->getSeconds() + $end->getNanos()/1000000000.0);
            printf('  Confidence: %f' . PHP_EOL, $shot->getConfidence());
        }
    }
    print(PHP_EOL);
} else {
    print_r($operation->getError());
}

Ruby

View on GitHub Feedback

# path = "Path to a video file on Google Cloud Storage: gs://bucket/video.mp4"

require "google/cloud/video_intelligence"

video = Google::Cloud::VideoIntelligence.video_intelligence_service

# Register a callback during the method call
operation = video.annotate_video features: [:LABEL_DETECTION], input_uri: path
puts "Processing video for label annotations:"
operation.wait_until_done!

raise operation.results.message? if operation.error?
puts "Finished Processing."

labels = operation.results.annotation_results.first.segment_label_annotations
print_labels labels