Label images by using the API in the Console

This quickstart walks you through the process of:

  • Creating a dataset of images with labeled bounding boxes.
  • Train a custom model using your dataset.

In this quickstart you use the custom user interface (UI) to interact with the AutoML Vision API. You can also complete all the steps in this quickstart using the AutoML API. For further directions on using the UI or API, see the how-to guides.

Before you begin

Before you can use AutoML Vision Object Detection, you must create a Google Cloud project (one never used with any other AutoML product) and enable AutoML Vision Object Detection for that project.

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Cloud AutoML and Storage APIs.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Cloud AutoML and Storage APIs.

    Enable the APIs

Beta requirements

  1. This Beta requires you to use us-central1 as the specified region.

Preparing a dataset

In this quickstart you use a dataset created from Open Images Dataset V4. This publicly available Salads dataset is located at gs://cloud-ml-data/img/openimage/csv/salads_ml_use.csv.

The CSV format is as follows:

TRAINING,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Salad,0.0,0.0954,,,0.977,0.957,,
VALIDATION,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Seafood,0.0154,0.1538,,,1.0,0.802,,
TEST,gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg,Tomato,0.0,0.655,,,0.231,0.839,,
dataset image example
3916261642_0a504acd60_o.jpg

Each row corresponds to an object localized inside a larger image, with each object specifically designated as test, train, or validation data. The three lines included here indicate three distinct objects located inside the same image available at gs://cloud-ml-data/img/openimage/3/2520/3916261642_0a504acd60_o.jpg. Each row has a different label: Salad, Seafood, Tomato, in addition to other rows with Baked goods or Cheese labels.

Bounding boxes are specified for each image using the top left and bottom right vertices:

  • (0,0) corresponds to the top left-most vertex.
  • (1,1) corresponds to the bottom right-most vertex.

For the first row shown above, the (x, y) coordinates for the top left vertex of the Salad labeled object are (0.0,0.0954), and the coordinates for the bottom right vertex of the object are (0.977,0.957).

For more detailed information on how to format your CSV file and the minimum requirements for creating a valid dataset, see Preparing your training data.


Create a dataset and import training images

  1. Open the AutoML Vision Object Detection UI and select your project from the drop-down list in the title bar.

    The first time you open the AutoML Vision Object Detection UI you will need to Enable AutoML API if it isn't already enabled.

  2. Select Get started in the pop-up window when prompted.

    Select get started option

  3. You will be taken to the datasets list page. Create a new dataset by selecting New Dataset.

    Select create new dataset

    Enter a unique name for your dataset.

    Select create new dataset's name

  4. Enter the location of the training data to import in the window that follows.

    In the Select a CSV file on Cloud Storage text box, enter the path for the sample CSV file (the gs:// prefix is added automatically):

    cloud-ml-data/img/openimage/csv/salads_ml_use.csv

    Alternatively, you can select Browse and navigate to the CSV file in one of your Google Cloud Storage buckets.

    This quickstart uses sample data staged in a public Google Cloud Storage bucket. The training data are sample JPG images annotated with the bounding box and label of the objects you want the model to learn to identify. To import the training data into the dataset, use a CSV file that points to the image (JPEG, PNG, GIF, BMP, or ICO) files; see Preparing your training data for information about the format and image specifications.

    Creating a dataset upload CSV image
  5. Select Import.

    Your dataset will show a Status of Running:importing images while your images are being imported. This process takes just a few minutes.

When your training data has been successfully imported, the Status column says Success:Creating dataset and the UI shows the generated ID for the dataset (used when making AutoML API calls) as well as the number of items imported.

Listing dataset image

If there are any problems importing images you will see Warning: Importing images as the status. Select the dataset name and Details to see the errors with specific image import.

Import error image

Train your model

After you have created your dataset and imported your training data into your dataset, you can train your custom model.

  1. From the Datasets listing page, click the dataset name.

    Listing a dataset page

  2. Select the Train tab. This will show you all labels and their division into train, test, and validation sets.

    Train option with train, test, validation sets

  3. Select Start training to open a side panel with training options.

    training panel

  4. Enter a name for your custom model or accept the default name. In this panel you can also select model optimization (for latency or accuracy).

    Select Deploy model to node after training to opt-in to automatic deployment. Otherwise, you will be prompted to manually deploy your model after training before you are able to make predictions.

  5. Select Start Training.

    Start model training page

Training a model can take several hours to complete. Typical training time for the sample is about one hour.

After the model is successfully trained, you will receive a message at the email address you used to sign up for the program.

Evaluate the custom model

After training a model, AutoML Vision Object Detection evaluates the quality and accuracy of the new model. To see the evaluation metrics for your model:

  1. Open the AutoML Vision Object Detection UI and click the Models tab (with lightbulb icon) in the left navigation bar.

  2. Click the name of the model you want to evaluate.

  3. If necessary, click the Evaluate tab just below the title bar.

    If training has been completed for the model, AutoML Vision Object Detection shows its evaluation metrics. It provides precision and recall scores for the model as a whole and for each object label at different score and intersection over union (IoU) thresholds. To see the metrics for a particular object label, choose the label from the "Filter labels" list.

    Model evaluation page

Precision and recall measure how well the model is capturing information, and how much it's leaving out. Precision indicates, from all objects annotated with a specific label, how many actually were supposed to be assigned to that label. Recall indicates, from all the objects that should have been identified as a particular entity, how many were actually assigned to that entity.

Use this data to evaluate your model's readiness:

  • Low precision or recall scores can indicate that your model needs additional training data.
  • Perfect precision and recall can indicate that the data is too easy and may not generalize well.

Use the custom model

After your model has been successfully trained, you can use it to identify objects in an image with bounding boxes and labels using your custom model. Select the Test and Use tab.

If you didn't opt-in for auto-deploy you will be prompted to deploy your model before you can make predictions.

Make a prediction

After your model has been deployed, indicate the path to your test image on the Test and use page (such as this image, saved locally). AutoML Vision analyzes the image using your model and displays labeled objects and their bounding boxes in the image.

Prediction on uploaded image page

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

If you no longer need your custom model or dataset, you can delete them.

To avoid unnecessary Google Cloud Platform charges, use the GCP Console to delete your project if you do not need it.

Undeploy your model

Your model incurs charges while it is deployed.

  1. Select the Test & Use tab just below the title bar.
  2. Select Remove deployment from the banner beneath your model name to open the undeploy option window.

    undeploy popup menu

  3. Select Remove deployment to undeploy the model.

    model deploying

  4. You will receive an email when model undeployment has completed.

Delete your project (optional)

To avoid unnecessary Google Cloud Platform charges, use the Google Cloud console to delete your project if you do not need it.

What's next