Getting started with the built-in image object detection algorithm

With built-in algorithms on AI Platform Training, you can submit your training data, select an algorithm, and let AI Platform Training handle the preprocessing and training for you, without writing any code for a training application. Built-in image algorithms allow you to train on TPUs with minimal configuration. The resulting TensorFlow SavedModel is compatible for serving on CPUs and GPUs.

Overview

In this tutorial, you train an image object detection model without writing any code. You submit the COCO dataset to AI Platform Training for training, and then you deploy the model on AI Platform Training to get predictions. The resulting model classifies common objects within images of complex everyday scenes.

Before you begin

To complete this tutorial on the command line, use either Cloud Shell or any environment where the Google Cloud CLI is installed.

Complete the following steps to set up a GCP account, enable the required APIs, and install and activate the Google Cloud CLI:

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the AI Platform Training & Prediction and Compute Engine APIs.

    Enable the APIs

  5. Install the Google Cloud CLI.

  6. To initialize the gcloud CLI, run the following command: 

    gcloud init
    7.

  7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  8. Make sure that billing is enabled for your Google Cloud project.

  9. Enable the AI Platform Training & Prediction and Compute Engine APIs.

    Enable the APIs

  10. Install the Google Cloud CLI.

  11. To initialize the gcloud CLI, run the following command: 

    gcloud init
    12.

Authorize your Cloud TPU to access your project

Follow these steps to authorize the Cloud TPU service account name associated with your Google Cloud project:

  1. Get your Cloud TPU service account name by calling projects.getConfig. Example:

    PROJECT_ID=PROJECT_ID

curl -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
    https://ml.googleapis.com/v1/projects/$PROJECT_ID:getConfig

  2. Save the value of the serviceAccountProject and tpuServiceAccount field returned by the API.

  3. Initialize the Cloud TPU service account:

    curl -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
  -H "Content-Type: application/json" -d '{}'  \
  https://serviceusage.googleapis.com/v1beta1/projects/<serviceAccountProject>/services/tpu.googleapis.com:generateServiceIdentity

Now add the Cloud TPU service account as a member in your project, with the role Cloud ML Service Agent. Complete the following steps in the Google Cloud console or using the gcloud command:

Console

  1. Log in to the Google Cloud console and choose the project in which you're using the TPU.
  2. Choose IAM & Admin > IAM.
  3. Click the Add button to add a member to the project.
  4. Enter the TPU service account in the Members text box.
  5. Click the Roles dropdown list.
  6. Enable the Cloud ML Service Agent role (Service Agents > Cloud ML Service Agent).

gcloud

  1. Set environment variables containing your project ID and the Cloud TPU service account:

    PROJECT_ID=PROJECT_ID
SVC_ACCOUNT=your-tpu-sa-123@your-tpu-sa.google.com.iam.gserviceaccount.com

  2. Grant the ml.serviceAgent role to the Cloud TPU service account:

    gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SVC_ACCOUNT --role roles/ml.serviceAgent

For more details about granting roles to service accounts, see the IAM documentation.

Setup

We have modified the COCO dataset for use with this tutorial, and hosted it in a public Cloud Storage bucket, gs://cloud-samples-data/ai-platform/built-in/image/coco/.

Console

Select your algorithm

  1. Go to the AI Platform Training Jobs page in the Google Cloud console:

    AI Platform Training Jobs page

  2. Click the New training job button. From the options that display below, click Built-in algorithm training. The Create a new training job page displays.

  3. The training job creation is divided into four steps. The first step is Training algorithm. Select image object detection and click Next.

Training data

  1. In the Training data section, select the training data for the sample dataset, hosted in our public Cloud Storage bucket:

    1. Select Use multiple files stored in one Cloud Storage directory.

    2. For Directory path, fill in: "cloud-samples-data/ai-platform/built-in/image/coco/"

    3. For Wildcard name, fill in "train*" to select all the training files in the directory.

    4. The Complete GCS path displays below: "gs://cloud-samples-data/ai-platform/built-in/image/coco/train*"

  2. In the Validation data section, select the validation data for the sample dataset, hosted in our public Cloud Storage bucket:

    1. Select Use multiple files stored in one Cloud Storage directory.

    2. For Directory path, fill in: "cloud-samples-data/ai-platform/built-in/image/coco/"

    3. For Wildcard name, fill in "val*" to select all the validation files in the directory.

    4. The Complete GCS path displays below: "gs://cloud-samples-data/ai-platform/built-in/image/coco/val*"

  3. Specify the Output directory in your Cloud Storage bucket where you want AI Platform Training to store your trained model, checkpoints, and other training job output. You can fill in the exact path in your bucket, or use the Browse button to select the path.

gcloud

Set up environment variables for your project ID, your Cloud Storage bucket, the Cloud Storage path to the training data, and your algorithm selection.

AI Platform Training built-in algorithms are in Docker containers hosted in Container Registry.

PROJECT_ID="YOUR_PROJECT_ID"
BUCKET_NAME="YOUR_BUCKET_NAME"
REGION="us-central1"

gcloud config set project $PROJECT_ID
gcloud config set compute/region $REGION

# Set paths to the training and validation data.
TRAINING_DATA_PATH="gs://cloud-samples-data/ai-platform/built-in/image/coco/train*"
VALIDATION_DATA_PATH="gs://cloud-samples-data/ai-platform/built-in/image/coco/val*"

# Specify the Docker container for your built-in algorithm selection.
IMAGE_URI="gcr.io/cloud-ml-algos/image_object_detection:latest"

Submit a training job

To submit a job, you must specify some basic training arguments and some basic arguments related to the image object detection algorithm.

General arguments for the training job:

Training job arguments
Argument Description
job-id Unique ID for your training job. You can use this to find logs for the status of your training job after you submit it.
job-dir Cloud Storage path where AI Platform Training saves training files after completing a successful training job.
scale-tier Specifies machine types for training. Use BASIC to select a configuration of just one machine.
master-image-uri Container Registry URI used to specify which Docker container to use for the training job. Use the container for the built-in image object detection algorithm defined earlier as IMAGE_URI.
region Specify the available region in which to run your training job. For this tutorial, you can use the region us-central1.

Arguments specific to the built-in image object detection algorithm:

Algorithm arguments
Argument Description
training_data_path Path to a TFRecord path pattern used for training.
validation_data_path Path to a TFRecord path pattern used for validation.
pretrained_checkpoint_path The path of pretrained checkpoints. You may use some published checkpoints.
num_classes The number of classes in the training/validation data.
max_steps The number of steps that the training job will run.
train_batch_size The number of images to use per training step.
num_eval_images The number of total images used for evaluation.
If it is 0, all the images in validation_data_path will be used for evaluation.
learning_rate_decay_type The method by which the learning rate decays during training.
warmup_learning_rate The learning rate at the start of the warm-up phase.
warmup_steps The number of steps to run during the warm-up phase, or the length of the warm-up phase in steps. The training job uses warmup_learning_rate during the warm-up phase. When the warm-up phase is over, the training job uses initial_learning_rate.
initial_learning_rate The initial learning rate after the warm-up phase is complete.
stepwise_learning_rate_steps The steps to decay/change learning rates for stepwise learning rate decay type.
For example, 100,200 means the learning rate will change (with respect to stepwise_learning_rate_levels) at step 100 and step 200. Note that it will be respected only when learning_rate_decay_type is set to stepwise.
stepwise_learning_rate_levels The learning rate value of each step for stepwise learning rate decay type. Note that it will be respected only when learning_rate_decay_type is set to stepwise.
image_size The image size (width and height) used for training.
optimizer_type The optimizer used for training. It should be one of:
{momentum, adam, adadelta, adagrad, rmsprop}
optimizer_arguments The arguments for optimizer. It is a comma separated list of "name=value" pairs. It needs to be compatible with optimizer_type. Examples:
fpn_type The multi-level Feature Pyramid Network (FPN) type. Must be one of {fpn, nasfpn}.
resnet_depth The depth of ResNet backbone. Must be one of {18,34,50,101,152,200}.
max_num_bboxes_in_training The maximum number of proposed bounding boxes to use for training.
max_num_bboxes_in_prediction The maximum number of proposed bounding boxes to use in prediction outputs.
bbox_aspect_ratios The scale of size of the base anchors representing the aspect ratio anchors added on each level. The number indicates the ratio of width to height. For instance, "1.0,2.0,0.5" adds three anchors on each scale level.
nms_iou_threshold The threshold to decide whether bounding boxes overlap with respect to IOU for non-maximum suppression.
nms_score_threshold The threshold for deciding when to remove bounding boxes based on score.
focal_loss_alpha Focal loss alpha (balancing param) value.
focal_loss_gamma Focal loss gamma (focusing param) value.
aug_scale_min The minimum scale applied during image augmentation. Its value is between [0, 1.0].
aug_scale_max The maximum scale applied during image augmentation. Its value is between [1.0, inf].
aug_rand_hflip Boolean. If set to True, this setting augments training with random horizontal flip.

For a detailed list of all other image object detection algorithm flags, refer to the built-in image object detection reference.

Console

Algorithm arguments

In the first part of the Algorithm arguments tab, fill in the following values:

  • Number of classes: 91
  • Max steps: 15000
  • Train batch size: 64
  • Number of evaluation images: 5000

In the Model Section of the Algorithm arguments tab:

  1. For Pretrained checkpoint path, fill in gs://cloud-samples-data/ai-platform/built-in/image/pretrained_checkpoints/detection/.

  2. Leave all other fields on their default settings, and click Next.

Job settings

On the Job settings tab:

  1. Enter a unique Job ID (such as "object_detection_example").
  2. Enter an available region (such as "us-central1").
  3. To select machine types, select "CUSTOM" for the scale tier. A section to provide your Custom cluster specification displays.
    1. For the Master type, select complex_model_m.
    2. For the Worker type, select cloud_tpu. The worker count defaults to 1.

Click Done to submit the training job.

gcloud

  1. Set up all the arguments for the training job and the algorithm, before using gcloud to submit the job:

    DATASET_NAME="coco"
ALGORITHM="object_detection"
MODEL_NAME="${DATASET_NAME}_${ALGORITHM}_model"

# Give a unique name to your training job.
DATE="$(date '+%Y%m%d_%H%M%S')"
JOB_ID="${MODEL_NAME}_${DATE}"

# Make sure you have access to this Cloud Storage bucket.
JOB_DIR="gs://${BUCKET_NAME}/algorithms_training/${MODEL_NAME}/${DATE}"

  2. Submit the job:

    gcloud ai-platform jobs submit training $JOB_ID \
  --region=$REGION \
  --config=config.yaml \
  --job-dir=$JOB_DIR \
  -- \
  --training_data_path=$TRAINING_DATA_PATH \
  --validation_data_path=$VALIDATION_DATA_PATH \
  --train_batch_size=64 \
  --num_eval_images=500 \
  --train_steps_per_eval=2000 \
  --max_steps=15000 \
  --num_classes=90 \
  --warmup_steps=500 \
  --initial_learning_rate=0.08 \
  --fpn_type="nasfpn" \
  --aug_scale_min=0.8 \
  --aug_scale_max=1.2

  3. After the job is submitted successfully, you can view the logs using the following gcloud commands:

    gcloud ai-platform jobs describe $JOB_ID
gcloud ai-platform jobs stream-logs $JOB_ID

Understand your job directory

After the successful completion of a training job, AI Platform Training creates a trained model in your Cloud Storage bucket, along with some other artifacts. You can find the following directory structure within your JOB_DIR:

  • model/ (a TensorFlow SavedModel directory that also contains a deployment_config.yaml file)
    • saved_model.pb
    • deployment_config.yaml
  • eval/
    • events.out.tfevents.[timestamp].cmle-training-[timestamp]
    • events.out.tfevents...
    • ...
  • variables/
    • variables.data-00000-of-00001
    • variables.index

The job directory also contains various model checkpoint files.

Confirm that the directory structure in your JOB_DIR matches:

gcloud storage ls $JOB_DIR/* --all-versions

Deploy the trained model

AI Platform Training organizes your trained models using model and version resources. An AI Platform Training model is a container for the versions of your machine learning model.

To deploy a model, you create a model resource in AI Platform Training, create a version of that model, then use the model and version to request online predictions.

For more information on how to deploy models to AI Platform Training, see how to deploy a TensorFlow model.

Console

  1. On the Jobs page, you can find a list of all your training jobs. Click the name of the training job you just submitted ("object_detection" or the job name you used).

  2. On the Job details page, you can view the general progress of your job, or click View logs for a more detailed view of its progress.

  3. When the job is successful, the Deploy model button appears at the top. Click Deploy model.

  4. Select Deploy as new model, and enter a model name, such as "algorithms_object_detection_model". Next, click Confirm.

  5. On the Create version page, enter a version name, such as "v1", and leave all other fields at their default settings. Click Save.

gcloud

The training process with the built-in image object detection algorithm produces a file, deployment_config.yaml, that makes it easier to deploy your model on AI Platform Training for predictions.

  1. Copy the file to your local directory and view its contents:

    gcloud storage cp $JOB_DIR/model/deployment_config.yaml .
cat deployment_config.yaml

    Your deployment_config.yaml file should appear similar to the following:

    deploymentUri: gs://BUCKET_NAME/algorithms_training/coco_object_detection/model
framework: TENSORFLOW
labels:
  global_step: '1000'
  job_id: coco_object_detection_20190227060114
runtimeVersion: '1.14'

  2. Create the model and version in AI Platform Training:

    gcloud ai-platform models create $MODEL_NAME --regions $REGION

# Create a model and a version using the file above.
VERSION_NAME="v_${DATE}"

gcloud ai-platform versions create $VERSION_NAME \
  --model $MODEL_NAME \
  --config deployment_config.yaml

    The version takes a few minutes to create.

Get online predictions

When requesting predictions, you need to make sure that your input data is formatted as JSON.

  1. Download the training artifact files:

    gcloud storage cp $JOB_DIR/artifacts/* .

  2. Prepare the prediction input for one image.

    To send an online prediction request using the Google Cloud CLI, as in this example, write each instance to a row in a newline-delimited JSON file.

    Run the following commands in your terminal to create input for a single instance that you can send to AI Platform Prediction:

    The following Python script encodes a single image using base64, formats it for prediction, adds an instance key, and writes the result to a file named prediction_instances.json:

    import json
import base64
import tensorflow as tf

IMAGE_URI='gs://cloud-samples-data/ai-platform/built-in/image/tutorial_examples/coco_sample.jpeg'

with tf.gfile.Open(IMAGE_URI, 'rb') as image_file:
  encoded_string = base64.b64encode(image_file.read()).decode('utf-8')

image_bytes = {'b64': str(encoded_string)}
instances = {'image_bytes': image_bytes, 'key': '1'}
with open("prediction_instances.json","w") as f:
  f.write(json.dumps(instances))

  3. Send the prediction request:

    gcloud ai-platform predict --model $MODEL_NAME \
 --version $VERSION_NAME \
 --json-instances prediction_instances.json

For each object detected within the image, the prediction output includes classes, scores and the locations of bounding boxes.

About the data

The Microsoft Common Objects in Context (COCO) dataset is a large-scale object detection, segmentation, and captioning dataset.

What's next

Next
Training using the built-in image object detection algorithm