Label images by using the AutoML Vision API

This quickstart walks you through the process of:

  • Copying a CSV listing images and bounding boxes with labels into Google Cloud Storage.
  • Using AutoML Vision Object Detection to create your dataset, and train and deploy your model.

In this quickstart you use cURL commands to send requests to AutoML Vision API. You can also complete all the steps listed here in the user interface by referencing the Quickstart using the user interface. For further directions on using the UI or API, see the how-to guides.

Before you begin

Set up your project

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. Install the Google Cloud CLI.

  3. To initialize the gcloud CLI, run the following command: 

    gcloud init

  4. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  5. Make sure that billing is enabled for your Google Cloud project.

  6. Enable the AutoML and Cloud Storage APIs: 

    gcloud services enable storage-component.googleapis.com automl.googleapis.com storage-api.googleapis.com
    7.
  7. Install the Google Cloud CLI.

  8. To initialize the gcloud CLI, run the following command: 

    gcloud init

  9. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  10. Make sure that billing is enabled for your Google Cloud project.

  11. Enable the AutoML and Cloud Storage APIs: 

    gcloud services enable storage-component.googleapis.com automl.googleapis.com storage-api.googleapis.com
    12.
  12. Set the PROJECT_ID environment variable to your Project ID. 
    export PROJECT_ID=PROJECT_ID
    The AutoML API calls and resource names include your Project ID in them. The PROJECT_ID environment variable provides a convenient way to specify the ID.

Preparing a dataset

In this quickstart you use a dataset created from Open Images Dataset V4. This publicly available Salads dataset is located at gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv.

The CSV format is as follows:

TRAINING,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Salad,0.0,0.0954,,,0.977,0.957,,
VALIDATION,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Seafood,0.0154,0.1538,,,1.0,0.802,,
TEST,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Tomato,0.0,0.655,,,0.231,0.839,,
dataset image example
3916261642_0a504acd60_o.jpg

Each row corresponds to an object localized inside a larger image, with each object specifically designated as test, train, or validation data. The three lines included here indicate three distinct objects located inside the same image available at gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg. Each row has a different label: Salad, Seafood, Tomato, in addition to other rows with Baked goods or Cheese labels.

Bounding boxes are specified for each image using the top left and bottom right vertices:

  • (0,0) corresponds to the top left-most vertex.
  • (1,1) corresponds to the bottom right-most vertex.

For the first row shown above, the (x, y) coordinates for the top left vertex of the Salad labeled object are (0.0,0.0954), and the coordinates for the bottom right vertex of the object are (0.977,0.957).

For more detailed information on how to format your CSV file and the minimum requirements for creating a valid dataset, see Preparing your training data.


Create a dataset and import training data

The curl command uses the gcloud auth print-access-token command to obtain an access token for your Google Account.

For the Beta release use us-central1 as the region, regardless of your actual location.

Create a dataset

Use the following curl command to create a new dataset with a display name of your choosing:

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
"https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets" -d '{
    "display_name": "DISPLAY_NAME",
    "image_object_detection_dataset_metadata": {
    }
}'

The response returns a relative dataset ID (such as IOD5491013845671477445) that you will need in following steps.

{
  "name": "projects/PROJECT_ID/locations/us-central1/datasets/IOD5491013845671477445",
  "displayName": "DISPLAY_NAME",
  "createTime": "2018-10-29T15:45:53.353442Z",
  "imageObjectDetectionDatasetMetadata": {}
}

Import data

Import your training data into your dataset. The importData command takes as input the URI of your dataset. Here you will provide the publicly available Google Cloud Storage address where the Salads dataset is stored. This process may take up to 30 minutes.

  • Replace DATASET_ID with the dataset identifier for your dataset (not the display name). For example: IOD5491013845671477445.
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID:importData \
-d '{
  "input_config": {
    "gcs_source": {
       "input_uris": [
         "gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv"
        ]
    }
  }
}'

The response returns a relative operation ID (for example, IOD1555149246326374411) that can be used to get the status of the operation.

{
  "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD1555149246326374411",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-29T15:56:29.176485Z",
    "updateTime": "2018-10-29T15:56:29.176485Z",
    "importDataDetails": {}
  }
}

Get the status of the import operation

You can query the status of your import data operation by using the following curl command.

  • Replace OPERATION_ID with the operation id returned from the import data operation.
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID

The import operation can take some time to complete. When the import task is finished, you will see done: true in the status of the operation with no errors listed, as shown in the following example.

This request will also return any warnings for your dataset import. Errors were added to this dataset to show you an example of these warnings. Using this operation to see these warnings is a useful way to find errors in your data.

{
  "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD1555149246326374411",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-29T15:56:29.176485Z",
    "updateTime": "2018-10-29T16:10:41.326614Z",
    "importDataDetails": {}
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

Get a list of datasets

You can get a list of your datasets by using the following command.

curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets

You should see output similar to the following:

{
  "datasets": [
    {
      "name": "projects/PROJECT_NUMBER/locations/us-central1/datasets/DATASET_ID1",
      "displayName": "DISPLAY_NAME1",
      "createTime": "2018-10-29T15:45:53.353442Z",
      "exampleCount": 227,
      "imageObjectDetectionDatasetMetadata": {}
    },
    {
      "name": "projects/PROJECT_NUMBER/locations/us-central1/datasets/DATASET_ID2",
      "displayName": "DISPLAY_NAME2",
      "createTime": "2018-10-24T21:06:05.390059Z",
      "exampleCount": 227,
      "imageObjectDetectionDatasetMetadata": {}
    }
  ]
}

Train your model

Launch a model training application

After you have created your dataset and imported your training data into your dataset, you can train your custom model.

Train your model by using the following curl command.

  • Replace DATASET_ID with the dataset identifier for your dataset (not the display name).
  • Replace DISPLAY_NAME with a name that you choose for your model.
  • Specify image_object_detection_model_metadata.model_type. Two available options are to optimize for latency or accuracy:
    • cloud-low-latency-1 - Optimizes training for latency.
    • cloud-high-accuracy-1 - Optimizes training for accuracy.
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models -d '{
  "datasetId": "DATASET_ID",
  "displayName": "DISPLAY_NAME",
  "image_object_detection_model_metadata": {},
}'

You should receive an operation id for your train model operation (for example, IOD5644417707978784777) which you can use to get the status of the training operation.

{
  "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD5644417707978784777",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-29T16:41:23.902167Z",
    "updateTime": "2018-10-29T16:41:23.902167Z",
    "createModelDetails": {}
  }
}

The training process may take several hours to complete.

Get the status of the model training operation

You can query the status of your model training operation by using the following curl command.

  • Replace OPERATION_ID with the operation id for your training operation.
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID

You should see output similar to the following. When the operation is complete, you will see done: true with no errors listed.


{
  "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD5644417707978784777",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata",
    "createTime": "2018-10-24T22:08:23.327323Z",
    "updateTime": "2018-10-24T23:41:18.452855Z",
    "createModelDetails": {}
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.automl.v1beta1.Model",
    "name": "projects/PROJECT_NUMBER/locations/us-central1/models/IOD5644417707978784777"
  }
}

Verify that the model is available

After your model training operation successfully completes, you can verify that your model is available by using the following command to list the models for your project.

curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models

You should receive a model id (for example, IOD5644417707978784778) which you can use to get model evaluation metrics when the model is done training. You should see output similar to the following:

{
  "model": [
    {
      "name": "projects/PROJECT_NUMBER/locations/us-central1/models/IOD5644417707978784778",
      "displayName": "DISPLAY_NAME",
      "datasetId": "IOD5491013845671477445",
      "createTime": "2018-10-24T23:37:00.858493Z",
      "updateTime": "2018-10-24T23:37:00.858493Z",
      "deploymentState": "DEPLOYED",
      "imageObjectDetectionModelMetadata": {
          "modelType": "cloud-low-latency-1",
          "nodeCount": "1",
          "nodeQps": 1.2987012987012987
      }
    }
  ]
}

Evaluate the model

After your model has finished training you can list model evaluation metrics using the following curl command.

  • Replace MODEL_ID with the identifier for your model.
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID/modelEvaluations

AutoML Vision Object Detection provides an aggregate set of evaluation metrics indicating how well the model performs overall, as well as evaluation metrics for each label, indicating how well the model performs for that label under differing thresholds.

Deploy your model

Before you can make a prediction you must manually deploy your model.

Use the following command to deploy your model:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "x-goog-user-project: PROJECT_ID" \
  -H "Content-Type: application/json" \
  https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:deploy \
  -d '{
      "imageObjectDetectionModelDeploymentMetadata": {
        "nodeCount": 2
      }
    }'

Make a prediction

Send a prediction request

You can use your deployed model to make a prediction on a local image using the following JSON file and curl command.

  • Create a request JSON file called predict_request.json and provide a base64-encoded image in the "imageBytes" field.

predict_request.json 

{
    "payload": {
        "image": {
            "image_bytes": "/9j/4QAYRXhpZgAA...base64-encoded-image...9tAVx/zDQDlGxn//2Q=="
        }
    }
}

  • Replace MODEL_ID with the identifier for your model.
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
"https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:predict" -d @PATH_TO_JSON_FILE/predict_request.json

A successful request returns a response with one or more bounding boxes specified by two diagonally opposite normalizedVertices. Each bounding box identified has an associated confidence score and annotation (displayName).

{
  "payload": [
    {
      "imageObjectDetection": {
        "boundingBox": {
          "normalizedVertices": [
            {
              "x": 0.034553755,
              "y": 0.015524037
            },
            {
              "x": 0.941527,
              "y": 0.9912563
            }
          ]
        },
        "score": 0.9997793
      },
      "displayName": "Salad"
    },
    {
      "imageObjectDetection": {
        "boundingBox": {
          "normalizedVertices": [
            {
              "x": 0.11737197,
              "y": 0.7098793
            },
            {
              "x": 0.510878,
              "y": 0.87987
            }
          ]
        },
        "score": 0.63219965
      },
      "displayName": "Tomato"
    }
  ]
}

Undeploy your model (optional)

Your model incurs charges while it is deployed. To avoid incurring this model-hosting cost you can undeploy your model.

Run the following command to undeploy your model:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "x-goog-user-project: PROJECT_ID" \
  -H "Content-Type: application/json" \
  "https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:undeploy"

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, delete the Google Cloud project with the resources.

If you no longer need your custom model and the related dataset, you can delete them.

List models

You can list the models for your project, along with their identifiers, by using the following command:

curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models

Delete a model

You can delete a model by using the following command.

  • Replace MODEL_ID with the identifier for your model.
curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID

List datasets

You can list the datasets for your project, along with their identifiers, by using the following command:

curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets

Delete a dataset

You can delete a dataset by using the following command.

  • Replace DATASET_ID with the identifier for your model.
curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: PROJECT_ID" \
-H "Content-Type: application/json" \
https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID

What's next