Tracking objects in videos

Beta

This product is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products are available "as is" and might have limited support. For more information, see the launch stage descriptions.

After you have created (trained) a model, you can request a prediction for one or more videos using the batchPredict method. You supply a CSV file with a list of videos to the batchPredict method. The batchPredict method annotates videos by detecting and tracking objects that your model predicts.

The maximum lifespan for a model is two years. After two years, you must create and train a new model to continue annotating your videos.

Note: Only the us-central1 location is supported.

Predict Example

To request a batch of predictions from AutoML Video Object Tracking, create a CSV file that lists the Cloud Storage paths to the videos that you want to annotate. You can also specify a start and end time to tell AutoML Video Object Tracking to only annotate a segment of the video. The start time must be zero or greater and must be before the end time. The end time must be greater than the start time and less than or equal to the duration of the video.

The following example shows how to annotate an entire video by specifying a start and end time as 0,inf.

gs://my-videos-vcm/cow_video.mp4,0,inf
gs://my-videos-vcm/bird_video.mp4,10.00000,15.50000

You must also specify an output file path where AutoML Video Object Tracking should write the results of the predictions from your model. This path must be a Cloud Storage bucket and object where you have write permissions.

Each video can be up to 3 hours in duration with a maximum file size of 50GB. AutoML Video Object Tracking can generate predictions for approximately 100 hours of video in 12 hours of processing time.

Web UI

1. Open the AutoML Video Object Tracking UI.
2. Click the model that you want to use from the displayed list.
3. On the Test & Use tab for the model, do the following:
   • Under Test your model, select a CSV file to use for the prediction. The CSV file must provide a list of videos that you want to annotate.

   • Also under Test your model, select a directory within your Cloud Storage bucket to receive the annotation results.

     You may actually want to create a specific 'results' folder in your Cloud Storage bucket to hold the annotation results.

   • Click Get Predictions.
   

The process for getting predictions can take some time, depending on the number of videos that you want annotated.

When the process has completed, the results appear on the page for the model under Recent Predictions. To view the results, do the following:

1. Under Recent Predictions in the Predictions column, click View for the prediction you want to look at.
2. Under Video, select the name of the video you want to see the results for.

REST

Before using any of the request data, make the following replacements:

• model-id: replace with the ID of your model. The ID is the last element of the name of your model. For example, if the name of your model is projects/project-number/locations/location-id/models/VOT6312181905852727296, then the ID of your model is VOT6312181905852727296.
• request-id: assign any digital value to this field.
• bucket-name: replace with the name of your Cloud Storage bucket. For example: my-project-vcm.
• input-file: replace with the name of your CSV file that identifies the videos to annotate.
• output-storage-path: replace with the path to the Cloud Storage bucket where the output of the predictions are to be stored. AutoML Video Object Tracking creates a sub-folder for the results in this path named using the following format: prediction-model_name-timestamp. The sub-folder will contain a prediction file for each video in the batch request. You must have write permissions to this path.
• Note:
  • project-number: number of your project
  • location-id: the Cloud region where annotation should take place. Supported cloud regions are: us-east1, us-west1, europe-west1, asia-east1. If no region is specified, a region will be determined based on video file location.

HTTP method and URL:

POST https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/models/model-id:batchPredict

Request JSON body:

{
  "request_id": "request-id",
  "inputConfig": {
    "gcsSource": {
      "inputUris": ["gs://bucket-name/input-file.csv"]
    }
  },
  "outputConfig": {
    "gcsDestination": {
      "outputUriPrefix": "gs://output-storage-path"
    }
  },
  "params": {
    "score_threshold": "0.0"
  }
}

To send your request, choose one of these options:

curl

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login , or by using Cloud Shell, which automatically logs you into the gcloud CLI . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: project-number" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/models/model-id:batchPredict"

PowerShell

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-number" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/models/model-id:batchPredict" | Select-Object -Expand Content

You should receive an operation ID for your model training operation. The example below shows a response that contains the model training operation ID VOT1741767155885539328.

Response

{
  "name": "projects/project-number/locations/location-id/operations/VOT1741767155885539328",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2020-02-18T21:32:06.730258Z",
    "updateTime": "2020-02-18T21:32:06.730258Z",
    "createModelDetails": {}
  }
}

Java

To authenticate to AutoML Video Object Tracking, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import com.google.api.gax.longrunning.OperationFuture;
import com.google.cloud.automl.v1beta1.BatchPredictInputConfig;
import com.google.cloud.automl.v1beta1.BatchPredictOutputConfig;
import com.google.cloud.automl.v1beta1.BatchPredictRequest;
import com.google.cloud.automl.v1beta1.BatchPredictResult;
import com.google.cloud.automl.v1beta1.GcsDestination;
import com.google.cloud.automl.v1beta1.GcsSource;
import com.google.cloud.automl.v1beta1.ModelName;
import com.google.cloud.automl.v1beta1.OperationMetadata;
import com.google.cloud.automl.v1beta1.PredictionServiceClient;
import java.io.IOException;
import java.util.concurrent.ExecutionException;

abstract class BatchPredict {

  static void batchPredict() throws IOException, ExecutionException, InterruptedException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String inputUri = "gs://YOUR_BUCKET_ID/path_to_your_input_csv_or_jsonl";
    String outputUri = "gs://YOUR_BUCKET_ID/path_to_save_results/";
    batchPredict(projectId, modelId, inputUri, outputUri);
  }

  static void batchPredict(String projectId, String modelId, String inputUri, String outputUri)
      throws IOException, ExecutionException, InterruptedException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (PredictionServiceClient client = PredictionServiceClient.create()) {
      // Get the full path of the model.
      ModelName name = ModelName.of(projectId, "us-central1", modelId);

      // Configure the source of the file from a GCS bucket
      GcsSource gcsSource = GcsSource.newBuilder().addInputUris(inputUri).build();
      BatchPredictInputConfig inputConfig =
          BatchPredictInputConfig.newBuilder().setGcsSource(gcsSource).build();

      // Configure where to store the output in a GCS bucket
      GcsDestination gcsDestination =
          GcsDestination.newBuilder().setOutputUriPrefix(outputUri).build();
      BatchPredictOutputConfig outputConfig =
          BatchPredictOutputConfig.newBuilder().setGcsDestination(gcsDestination).build();

      // Build the request that will be sent to the API
      BatchPredictRequest request =
          BatchPredictRequest.newBuilder()
              .setName(name.toString())
              .setInputConfig(inputConfig)
              .setOutputConfig(outputConfig)
              .build();

      // Start an asynchronous request
      OperationFuture<BatchPredictResult, OperationMetadata> future =
          client.batchPredictAsync(request);

      System.out.println("Waiting for operation to complete...");
      future.get();
      System.out.println("Batch Prediction results saved to specified Cloud Storage bucket.");
    }
  }
}

Node.js

To authenticate to AutoML Video Object Tracking, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
// const projectId = 'YOUR_PROJECT_ID';
// const location = 'us-central1';
// const modelId = 'YOUR_MODEL_ID';
// const inputUri = 'gs://YOUR_BUCKET_ID/path_to_your_input_csv_or_jsonl';
// const outputUri = 'gs://YOUR_BUCKET_ID/path_to_save_results/';

// Imports the Google Cloud AutoML library
const {PredictionServiceClient} = require('@google-cloud/automl').v1beta1;

// Instantiates a client
const client = new PredictionServiceClient();

async function batchPredict() {
  // Construct request
  const request = {
    name: client.modelPath(projectId, location, modelId),
    inputConfig: {
      gcsSource: {
        inputUris: [inputUri],
      },
    },
    outputConfig: {
      gcsDestination: {
        outputUriPrefix: outputUri,
      },
    },
  };

  const [operation] = await client.batchPredict(request);

  console.log('Waiting for operation to complete...');
  // Wait for operation to complete.
  const [response] = await operation.promise();
  console.log(
    `Batch Prediction results saved to Cloud Storage bucket. ${response}`
  );
}

batchPredict();

Python

To authenticate to AutoML Video Object Tracking, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

from google.cloud import automl_v1beta1 as automl


def batch_predict(
    project_id="YOUR_PROJECT_ID",
    model_id="YOUR_MODEL_ID",
    input_uri="gs://YOUR_BUCKET_ID/path/to/your/input/csv_or_jsonl",
    output_uri="gs://YOUR_BUCKET_ID/path/to/save/results/",
):
    """Batch predict"""
    prediction_client = automl.PredictionServiceClient()

    # Get the full path of the model.
    model_full_id = automl.AutoMlClient.model_path(project_id, "us-central1", model_id)

    gcs_source = automl.GcsSource(input_uris=[input_uri])

    input_config = automl.BatchPredictInputConfig(gcs_source=gcs_source)
    gcs_destination = automl.GcsDestination(output_uri_prefix=output_uri)
    output_config = automl.BatchPredictOutputConfig(gcs_destination=gcs_destination)
    params = {}

    request = automl.BatchPredictRequest(
        name=model_full_id,
        input_config=input_config,
        output_config=output_config,
        params=params,
    )
    response = prediction_client.batch_predict(request=request)

    print("Waiting for operation to complete...")
    print(
        "Batch Prediction results saved to Cloud Storage bucket. {}".format(
            response.result()
        )
    )

Get the status of the predict operation

You can query the status of your batch predict operation by using the following curl or PowerShell commands.

REST

Before using any of the request data, make the following replacements:

• project-number: the number of your project
• location-id: the Cloud region where annotation should take place. Supported cloud regions are: us-east1, us-west1, europe-west1, asia-east1. If no region is specified, a region will be determined based on video file location.
• operation-id: the ID of the long-running operation created for the request and provided in the response when you started the operation, for example VOT12345....

HTTP method and URL:

GET https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/operations/operation-id

To send your request, choose one of these options:

curl

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login , or by using Cloud Shell, which automatically logs you into the gcloud CLI . You can check the currently active account by running gcloud auth list.

Execute the following command:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: project-number" \
     "https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/operations/operation-id"

PowerShell

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login . You can check the currently active account by running gcloud auth list.

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-number" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://automl.googleapis.com/v1beta1/projects/project-number/locations/location-id/operations/operation-id" | Select-Object -Expand Content

The operation-name is the name of the operation as returned by the AutoML Video Intelligence Object Tracking API. The operation name has the format projects/project-number/locations/location-id/operations/operation-id

Response

{
  "name": "operation-name",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2020-02-18T21:23:15.181451Z",
    "updateTime": "2020-02-18T21:26:10.354502Z",
    "importDataDetails": {}
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

When the batch predict task is complete, the output of the prediction is stored in the Cloud Storage bucket that you specified in your command. There is a JSON file for each video segment. The JSON files have a format similar to my-video-01.avi.json, where file extension .json is appended to the original file name.

{
  "inputUris": ["automl-video-demo-data/sample_video.avi"],
  "object_annotations": [ {
    "annotation_spec": {
      "display_name": "Cat",
      "description": "Cat"
    },
    "confidence": 0.43253016
    "frames": [ {
      "frame": {
        "time_offset": {
          "seconds": 4,
          "nanos": 960000000
        },
        "normalized_bounding_box": {
          "x_min": 0.1,
          "y_min": 0.1,
          "x_max": 0.8,
          "y_max": 0.8
        }
      }
    }, {
      "frame": {
        "time_offset": {
          "seconds": 5,
          "nanos": 960000000
        },
        "normalized_bounding_box": {
          "x_min": 0.2,
          "y_min": 0.2,
          "x_max": 0.9,
          "y_max": 0.9
        }
      }
    } ],
    "segment": {
      "start_time_offset": {
          "seconds": 4,
          "nanos": 960000000
      },
      "end_time_offset": {
          "seconds": 5,
          "nanos": 960000000
      }
    }
  } ],
  "error": {
    "details": [ ]
  }
}