Annotating videos

After you have created (trained) a model, you can request a prediction for one or more videos using the batchPredict method. You supply a CSV file with a list of videos to the batchPredict method. The batchPredict method applies labels to your videos based on the primary objects of the videos that your model predicts.

The maximum lifespan for a custom model is two years. You must create and train a new model to continue classifying content after that amount of time.

Using curl

To make it more convenient to run the curl samples in this topic, set the following environment variable. Replace project-id with the name of your GCP project.

export PROJECT_ID="project-id"

Predict Example

To request a batch of predictions from AutoML Video, create a CSV file that lists the Google Cloud Storage paths to the videos that you want to annotate. You can also specify a start and end time to tell AutoML Video to only annotate a segment of the video. The start time must be zero or greater and must be before the end time. The end time must be greater than the start time and less than or equal to the duration of the video. You can also use inf to indicate the end of a video.

gs://my-videos-vcm/short_video_1.avi,0.0,5.566667
gs://my-videos-vcm/car_chase.avi,0.0,3.933333
gs://my-videos-vcm/northwest_wildlife_01.avi,0.0,3.7
gs://my-videos-vcm/northwest_wildlife_02.avi,0.0,1.666667
gs://my-videos-vcm/motorcycles_and_cars.avi,0.0,2.633333
gs://my-videos-vcm/drying_paint.avi,0.0,inf

You must also specify an output file path where AutoML Video will write the results of the predictions from your model. This path must be a Google Cloud Storage bucket and object where you have write permissions.

Each video can be up to 3 hours in duration with a maximum file size of 50GB. AutoML Video can generate predictions for approximately 100 hours of video in 12 hours of processing time.

Optionally, You can set the following options in the params section when you request a prediction for your videos. If you do not specify any of these options, then the default score threshold is applied, and segment-classification is used.

  • score_threshold - A value from 0.0 to 1.0. When the model makes predictions for a video, it will only produce results that have at least this confidence score. The default is 0.5.

  • segment-classification - Set to true to enable segment-level classification. AutoML Video returns labels and their confidence scores for the entire segment of video that you specified in your request configuration. The default is true.

  • shot-classification - Set to true to enable shot-level classification. AutoML Video determines the boundaries for each camera shot in the entire segment of video that you specified in your request configuration. AutoML Video Intelligence then returns labels and their confidence scores for each shot detected, along with the start and end time of the shot. The default is false.

  • 1s_interval - Set to true to enable classification for a video at one-second intervals. AutoML Video returns labels and their confidence scores for each second of the entire segment of video that you specified in your request configuration. The default is false.

Web UI

  1. Open the Google Cloud Platform Console.
  2. Click the custom model that you want to use from the displayed list. List of custom AutoML Video Intelligence models in the Cloud Console
  3. On the Predict tab for the model, do the following:
    • Under Test your model on new videos, select a CSV file to use for the prediction. The CSV file must provide a list of videos that you want to annotate.
    • Also under Test your model on new videos, select a directory within your Cloud Storage bucket to receive the annotation results.

      You may actually want to create a specific 'results' folder in your Cloud Storage bucket to hold the annotation results. By doing so, you can more easily access older predictions by loading the video_classification.csv file contained in the results directory.

    • Click Get Predictions.
    Configuring a prediction request for AutoML Video Intelligence

The process for getting predictions can take some time, depending on the number of videos that you want annotated.

When the process has completed, the results appear on the page for the model under Recent Predictions. To view the results, do the following:

  1. Under Recent Predictions in the Predictions column, click View for the prediction you want to look at.
  2. Under Video, select the name of the video you want to see the results for.

Results of AutoML Video Intelligence prediction

Command-line

The following example requests a prediction for a video.

  • Replace model-id with the ID of your model. The ID is the last element of the name of your model. For example, if the name of your model is projects/434039606874/locations/us-central1/models/3745331181667467569, then the ID of your model is 3745331181667467569.

  • Replace your-bucket-name with the name of your Google Cloud Storage bucket. For example: my-project-vcm.

  • Replace your-input-file with the name of your CSV file that identifies the videos to annotate.

  • Replace your-output-storage-path with a path to Google Cloud Storage where the output of the predictions are to be stored. AutoML Video will create a sub-folder in this path named with the following format: prediction-model_name-timestamp. The sub-folder will contain a prediction file for each video in the batch request. You must have write permissions to this path.

curl -X POST \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/us-central1/models/your-model-id:batchPredict -d '{
  "inputConfig": {
    "gcsSource": {
      "inputUris": ["gs://your-bucket-name/your-input-file.csv"]
    }
  },
  "outputConfig": {
    "gcsDestination": {
      "outputUriPrefix": "gs://your-storage-path"
    }
  }
}'

You should receive an operation id for your batch predict request. For example: VCN926615623331479552.

{
  "name": "projects/915378992997/locations/us-central1/operations/VCN926615623331479552",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-19T21:22:27.003694Z",
    "updateTime": "2018-10-19T21:22:27.003694Z",
    "batchPredictDetails": {}
  }
}

You can query the status of your batch predict operation by using the following curl command.

  • Replace your-operation-id with the operation id for your batch predict operation.
curl -X GET -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json" \
https://${ENDPOINT}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/operations/your-operation-id

Depending on the number of videos that you specified in your CSV file, the batch predict task can take some time to complete. When the task is finished, you will see done: true in the status of the operation with no errors listed, as shown in the following example.

{
  "name": "projects/915378992997/locations/us-central1/operations/VCN926615623331479552",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-18T21:23:15.181451Z",
    "updateTime": "2018-10-18T21:26:10.354502Z",
    "importDataDetails": {}
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

When the batch predict task is complete, the output of the prediction is stored in the Google Cloud Storage bucket that you specified in your command. There is a JSON file for each video segment. For example:

my-video-01.avi.json

{
  "input_uri": "automl-video-sample/sample_video.avi",
  "segment_classification_annotations": [ {
    "annotation_spec": {
      "display_name": "ApplyLipstick",
      "description": "ApplyLipstick"
    },
    "segments": [ {
      "segment": {
        "start_time_offset": {
        },
        "end_time_offset": {
          "seconds": 4,
          "nanos": 960000000
        }
      },
      "confidence": 0.43253016
    }, {
      "segment": {
        "start_time_offset": {
        },
        "end_time_offset": {
          "seconds": 4,
          "nanos": 960000000
        }
      },
      "confidence": 0.56746984
    } ],
    "frames": [ ]
  } ],
  "shot_classification_annotations": [ {
    "annotation_spec": {
      "display_name": "ApplyLipstick",
      "description": "ApplyLipstick"
    },
    "segments": [ {
      "segment": {
        "start_time_offset": {
        },
        "end_time_offset": {
          "seconds": 5
        }
      },
      "confidence": 0.43253016
    }, {
      "segment": {
        "start_time_offset": {
        },
        "end_time_offset": {
          "seconds": 5
        }
      },
      "confidence": 0.56746984
    } ],
    "frames": [ ]
  } ],
  "one_second_sliding_window_classification_annotations": [ {
    "annotation_spec": {
      "display_name": "ApplyLipstick",
      "description": "ApplyLipstick"
    },
    "segments": [ ],
    "frames": [ {
      "time_offset": {
        "nanos": 800000000
      },
      "confidence": 0.54533803
    }, {
      "time_offset": {
        "nanos": 800000000
      },
      ...
      "confidence": 0.57945728
    }, {
      "time_offset": {
        "seconds": 4,
        "nanos": 300000000
      },
      "confidence": 0.42054281
    } ]
  } ],
  "object_annotations": [ ],
  "error": {
    "details": [ ]
  }
}

Java

/**
 * Demonstrates using the AutoML client to classify the video intelligence
 *
 * @param projectId the Id of the project.
 * @param computeRegion the Region name. (e.g., "us-central1")
 * @param modelId the Id of the model which will be used for video intelligence classification.
 * @param inputUri the GCS bucket path of csv file which contains path of the video to be
 *     classified.
 * @param outputUriPrefix the output GCS bucket folder path which contains one csv file and json
 *     file for each video classification.
 * @throws IOException
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void predict(
    String projectId,
    String computeRegion,
    String modelId,
    String inputUri,
    String outputUriPrefix)
    throws IOException, InterruptedException, ExecutionException {

  // Create client for prediction service.
  PredictionServiceClient predictionClient = PredictionServiceClient.create();

  // Get full path of model
  ModelName modelName = ModelName.of(projectId, computeRegion, modelId);

  // Set the input URI
  GcsSource.Builder gcsSource = GcsSource.newBuilder();

  // Get multiple training data files to be imported
  String[] inputUris = inputUri.split(",");
  for (String inputFilePath : inputUris) {
    gcsSource.addInputUris(inputFilePath);
  }

  // Set the Batch Input Configuration
  BatchPredictInputConfig batchInputConfig =
      BatchPredictInputConfig.newBuilder().setGcsSource(gcsSource).build();

  // Set the output URI
  GcsDestination.Builder gcsDestination = GcsDestination.newBuilder();
  gcsDestination.setOutputUriPrefix(outputUriPrefix);

  // Set the Batch Input Configuration
  BatchPredictOutputConfig batchOutputConfig =
      BatchPredictOutputConfig.newBuilder().setGcsDestination(gcsDestination).build();

  // Set the modelName, input and output config in the batch prediction
  BatchPredictRequest batchRequest =
      BatchPredictRequest.newBuilder()
          .setInputConfig(batchInputConfig)
          .setOutputConfig(batchOutputConfig)
          .setName(modelName.toString())
          .build();

  // Get the latest state of a long-running operation.
  OperationFuture<BatchPredictResult, OperationMetadata> operation =
      predictionClient.batchPredictAsync(batchRequest);

  System.out.println(
      String.format("Operation Name: %s", operation.getInitialFuture().get().getName()));
}

Node.js

const automl = require(`@google-cloud/automl`);

// Create client for prediction service.
const client = new automl.v1beta1.PredictionServiceClient();

/**
 * Demonstrates using the AutoML client to classify the video intelligence
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// const projectId = '[PROJECT_ID]' e.g., "my-gcloud-project";
// const computeRegion = '[REGION_NAME]' e.g., "us-central1";
// const modelId = '[MODEL_ID]' e.g., "IOD2122286140026257408";
// const inputUri = '[GCS_PATH]' e.g., "gs://<bucket-name>/<csv file>",
// `the GCS bucket path of csv file which contains path of the video
// to be classified.`;
// const outputUriPrefix = '[GCS_DIRECTORY_PATH]'
// e.g., "gs://<bucket-name>/<folder>",
// `the output GCS bucket folder path which contains one csv file and
// json file for each video classification.`;

// Get the full path of the model.
const modelFullId = client.modelPath(projectId, computeRegion, modelId);

// Set the input URI
const inputConfig = {
  gcsSource: {
    inputUris: [inputUri],
  },
};

// Set the output URI
const outputUri = outputUriPrefix;
const outputConfig = {
  gcsDestination: {
    outputUriPrefix: outputUri,
  },
};

// Get the latest state of a long-running operation.
client
  .batchPredict({
    name: modelFullId,
    inputConfig: inputConfig,
    outputConfig: outputConfig,
  })
  .then(responses => {
    const response = responses[1];
    console.log(`Operation name: ${response.name}`);
  })
  .catch(err => {
    console.error(err);
  });

Python

# TODO(developer): Uncomment and set the following variables
# project_id = 'PROJECT_ID_HERE'
# compute_region = 'COMPUTE_REGION_HERE'
# model_id = 'MODEL_ID_HERE'
# input_uri = 'gs://path/to/file.csv'
# output_uri_prefix = 'gs://path'


from google.cloud import automl_v1beta1 as automl
import csv

automl_client = automl.AutoMlClient()

# Get the full path of the model.
model_full_id = automl_client.model_path(
    project_id, compute_region, model_id
)

# Create client for prediction service.
prediction_client = automl.PredictionServiceClient()

# Input configuration.
input_config = dict(gcs_source={'input_uris': [input_uri]})

# Output configuration.
output_config = dict(
    gcs_destination={'output_uri_prefix': output_uri_prefix}
)

# Launch long-running batch prediction operation.
response = prediction_client.batch_predict(model_full_id, input_config,
                                            output_config)
print("Making batch prediction... ")
try:
    result = response.result()
except:
    # Hides Any to BatchPredictResult error.
    pass
print("Batch prediction complete.\n{}".format(response.metadata))

Was this page helpful? Let us know how we did:

Send feedback about...

AutoML Video Intelligence