Label images by using the AutoML Vision API
This quickstart walks you through the process of:
- Copying a CSV listing images and bounding boxes with labels into Google Cloud Storage.
- Using AutoML Vision Object Detection to create your dataset, and train and deploy your model.
In this quickstart you use cURL commands to send requests to AutoML Vision API. You can also complete all the steps listed here in the user interface by referencing the Quickstart using the user interface. For further directions on using the UI or API, see the how-to guides.
Before you begin
Set up your project
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the AutoML and Cloud Storage APIs:
gcloud services enable storage-component.googleapis.com
automl.googleapis.com storage-api.googleapis.com - Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the AutoML and Cloud Storage APIs:
gcloud services enable storage-component.googleapis.com
automl.googleapis.com storage-api.googleapis.com - Set the
PROJECT_ID
environment variable to your Project ID.export PROJECT_ID=PROJECT_ID
The AutoML API calls and resource names include your Project ID in them. ThePROJECT_ID
environment variable provides a convenient way to specify the ID.
Preparing a dataset
In this quickstart you use a dataset created from Open Images Dataset V4.
This publicly available Salads dataset is located at
gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv
.
The CSV format is as follows:
TRAINING,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Salad,0.0,0.0954,,,0.977,0.957,, VALIDATION,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Seafood,0.0154,0.1538,,,1.0,0.802,, TEST,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Tomato,0.0,0.655,,,0.231,0.839,,
Each row corresponds to an object localized inside a larger image, with each
object specifically designated as test, train, or validation data. The three
lines included here indicate three distinct objects located inside the same
image available at
gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg
. Each row
has a different label: Salad
, Seafood
, Tomato
, in addition
to other rows with Baked goods
or Cheese
labels.
Bounding boxes are specified for each image using the top left and bottom right vertices:
- (0,0) corresponds to the top left-most vertex.
- (1,1) corresponds to the bottom right-most vertex.
For the first row shown above, the (x, y) coordinates for the top left vertex
of the Salad
labeled object are (0.0,0.0954), and the coordinates for the
bottom right vertex of the object are (0.977,0.957).
For more detailed information on how to format your CSV file and the minimum requirements for creating a valid dataset, see Preparing your training data.
Create a dataset and import training data
The curl
command uses the gcloud auth print-access-token
command to obtain an access token for your Google Account.
For the Beta release use us-central1
as the region, regardless of your actual
location.
Create a dataset
Use the following curl
command to create a new dataset with a display name of
your choosing:
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ "https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets" -d '{ "display_name": "DISPLAY_NAME", "image_object_detection_dataset_metadata": { } }'
The response returns a relative dataset ID (such as IOD5491013845671477445
) that you will need
in following steps.
{ "name": "projects/PROJECT_ID/locations/us-central1/datasets/IOD5491013845671477445", "displayName": "DISPLAY_NAME", "createTime": "2018-10-29T15:45:53.353442Z", "imageObjectDetectionDatasetMetadata": {} }
Import data
Import your training data into your dataset. The importData
command takes as
input the URI of your dataset. Here you will provide the publicly available
Google Cloud Storage address where the Salads dataset is stored. This process
may take up to 30
minutes.
- Replace DATASET_ID with the dataset identifier for your
dataset (not the display name). For example:
IOD5491013845671477445
.
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID:importData \ -d '{ "input_config": { "gcs_source": { "input_uris": [ "gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv" ] } } }'
The response returns a relative operation ID (for example, IOD1555149246326374411
)
that can be used to get the status of the operation.
{ "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD1555149246326374411", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata", "createTime": "2018-10-29T15:56:29.176485Z", "updateTime": "2018-10-29T15:56:29.176485Z", "importDataDetails": {} } }
Get the status of the import operation
You can query the status of your import data operation by using the following
curl
command.
- Replace OPERATION_ID with the operation id returned from the import data operation.
curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID
The import operation can take some time to complete. When the import task is
finished, you will see done: true
in the status of the operation with
no errors listed, as shown in the following example.
This request will also return any warnings for your dataset import. Errors were added to this dataset to show you an example of these warnings. Using this operation to see these warnings is a useful way to find errors in your data.
{ "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD1555149246326374411", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata", "createTime": "2018-10-29T15:56:29.176485Z", "updateTime": "2018-10-29T16:10:41.326614Z", "importDataDetails": {} }, "done": true, "response": { "@type": "type.googleapis.com/google.protobuf.Empty" } }
Get a list of datasets
You can get a list of your datasets by using the following command.
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets
You should see output similar to the following:
{ "datasets": [ { "name": "projects/PROJECT_NUMBER/locations/us-central1/datasets/DATASET_ID1", "displayName": "DISPLAY_NAME1", "createTime": "2018-10-29T15:45:53.353442Z", "exampleCount": 227, "imageObjectDetectionDatasetMetadata": {} }, { "name": "projects/PROJECT_NUMBER/locations/us-central1/datasets/DATASET_ID2", "displayName": "DISPLAY_NAME2", "createTime": "2018-10-24T21:06:05.390059Z", "exampleCount": 227, "imageObjectDetectionDatasetMetadata": {} } ] }
Train your model
Launch a model training application
After you have created your dataset and imported your training data into your dataset, you can train your custom model.
Train your model by using the following curl
command.
- Replace DATASET_ID with the dataset identifier for your dataset (not the display name).
- Replace DISPLAY_NAME with a name that you choose for your model.
- Specify
image_object_detection_model_metadata.model_type
. Two available options are to optimize for latency or accuracy:cloud-low-latency-1
- Optimizes training for latency.cloud-high-accuracy-1
- Optimizes training for accuracy.
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models -d '{ "datasetId": "DATASET_ID", "displayName": "DISPLAY_NAME", "image_object_detection_model_metadata": {}, }'
You should receive an operation id for your train model operation (for example,
IOD5644417707978784777
) which you
can use to get the status of the training operation.
{ "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD5644417707978784777", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata", "createTime": "2018-10-29T16:41:23.902167Z", "updateTime": "2018-10-29T16:41:23.902167Z", "createModelDetails": {} } }
The training process may take several hours to complete.
Get the status of the model training operation
You can query the status of your model training operation by using the following
curl
command.
- Replace OPERATION_ID with the operation id for your training operation.
curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID
You should see output similar to the following. When the operation is complete,
you will see done: true
with no errors listed.
{ "name": "projects/PROJECT_NUMBER/locations/us-central1/operations/IOD5644417707978784777", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata", "createTime": "2018-10-24T22:08:23.327323Z", "updateTime": "2018-10-24T23:41:18.452855Z", "createModelDetails": {} }, "done": true, "response": { "@type": "type.googleapis.com/google.cloud.automl.v1beta1.Model", "name": "projects/PROJECT_NUMBER/locations/us-central1/models/IOD5644417707978784777" } }
Verify that the model is available
After your model training operation successfully completes, you can verify that your model is available by using the following command to list the models for your project.
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models
You should receive a model id (for example,
IOD5644417707978784778
) which you
can use to get model evaluation metrics when the model is done training.
You should see output similar to the following:
{ "model": [ { "name": "projects/PROJECT_NUMBER/locations/us-central1/models/IOD5644417707978784778", "displayName": "DISPLAY_NAME", "datasetId": "IOD5491013845671477445", "createTime": "2018-10-24T23:37:00.858493Z", "updateTime": "2018-10-24T23:37:00.858493Z", "deploymentState": "DEPLOYED", "imageObjectDetectionModelMetadata": { "modelType": "cloud-low-latency-1", "nodeCount": "1", "nodeQps": 1.2987012987012987 } } ] }
Evaluate the model
After your model has finished training you can list model evaluation metrics
using the following curl
command.
- Replace MODEL_ID with the identifier for your model.
curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID/modelEvaluations
AutoML Vision Object Detection provides an aggregate set of evaluation metrics indicating how well the model performs overall, as well as evaluation metrics for each label, indicating how well the model performs for that label under differing thresholds.
Deploy your model
Before you can make a prediction you must manually deploy your model.
Use the following command to deploy your model:
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:deploy \ -d '{ "imageObjectDetectionModelDeploymentMetadata": { "nodeCount": 2 } }'
Make a prediction
Send a prediction request
You can use your deployed model to make a prediction on a local image
using the following JSON file and curl
command.
- Create a request JSON file called
predict_request.json
and provide a base64-encoded image in the"imageBytes"
field.
predict_request.json
{ "payload": { "image": { "image_bytes": "/9j/4QAYRXhpZgAA...base64-encoded-image...9tAVx/zDQDlGxn//2Q==" } } }
- Replace MODEL_ID with the identifier for your model.
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ "https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:predict" -d @PATH_TO_JSON_FILE/predict_request.json
A successful request returns a response with one or more bounding boxes
specified by two diagonally opposite normalizedVertices
.
Each bounding box identified has an associated confidence score
and
annotation (displayName
).
{ "payload": [ { "imageObjectDetection": { "boundingBox": { "normalizedVertices": [ { "x": 0.034553755, "y": 0.015524037 }, { "x": 0.941527, "y": 0.9912563 } ] }, "score": 0.9997793 }, "displayName": "Salad" }, { "imageObjectDetection": { "boundingBox": { "normalizedVertices": [ { "x": 0.11737197, "y": 0.7098793 }, { "x": 0.510878, "y": 0.87987 } ] }, "score": 0.63219965 }, "displayName": "Tomato" } ] }
Undeploy your model (optional)
Your model incurs charges while it is deployed. To avoid incurring this model-hosting cost you can undeploy your model.
Run the following command to undeploy your model:
curl -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ "https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID:undeploy"
Clean up
To avoid incurring charges to your Google Cloud account for the resources used on this page, delete the Google Cloud project with the resources.
If you no longer need your custom model and the related dataset, you can delete them.
List models
You can list the models for your project, along with their identifiers, by using the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models
Delete a model
You can delete a model by using the following command.
- Replace MODEL_ID with the identifier for your model.
curl -X DELETE \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/models/MODEL_ID
List datasets
You can list the datasets for your project, along with their identifiers, by using the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets
Delete a dataset
You can delete a dataset by using the following command.
- Replace DATASET_ID with the identifier for your model.
curl -X DELETE \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json" \ https://automl.googleapis.com/v1beta1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID