Running Deeplab-v3 on Cloud TPU

This tutorial shows you how to train the Deeplab-v3 model on Cloud TPU.

The instructions below assume you are already familiar with running a model on Cloud TPU. If you are new to Cloud TPU, you can refer to the Quickstart for a basic introduction.

If you plan to train on a TPU Pod slice, review Training on TPU Pods to understand parameter changes required for Pod slices.

This model is an image semantic segmentation model. Image semantic segmentation models focus on identifying and localizing multiple objects in a single image. This type of model is frequently used in machine learning applications such as autonomous driving, geospatial image processing, and medical imaging.

In this tutorial, you'll run a training model against the PASCAL VOC 2012 dataset. For more information on this data set, see The PASCAL Visual Object Classes Homepage.

Objectives

Create a Cloud Storage bucket to hold your dataset and model output.
Install the required packages.
Download and convert the PASCAL VOC 2012 dataset.
Train the Deeplab model.
Evaluate the Deeplab model.

Costs

This tutorial uses billable components of Google Cloud, including:

Compute Engine
Cloud TPU
Cloud Storage

Use the pricing calculator to generate a cost estimate based on your projected usage. New Google Cloud users might be eligible for a free trial.

Before you begin

This section provides information on setting up Cloud Storage bucket and a Compute Engine VM.

Open a Cloud Shell window.

Open Cloud Shell
Create a variable for your project's name.
```
export PROJECT_NAME=project-name
```
Configure gcloud command-line tool to use the project where you want to create Cloud TPU.
```
gcloud config set project ${PROJECT_NAME}
```
Create a Cloud Storage bucket using the following command:

Note: In the following command, replace bucket-name with the name you want to assign to your bucket.
```
gsutil mb -p ${PROJECT_NAME} -c standard -l us-central1 -b on gs://bucket-name
```
This Cloud Storage bucket stores the data you use to train your model and the training results.

In order for the Cloud TPU to read and write to the storage bucket, the service account for your project needs read/write or Admin permissions on it. See the section on storage buckets for how to view and set those permissions.
Launch a Compute Engine VM using the ctpu up command.
```
$ ctpu up --zone=us-central1-b \
 --machine-type=n1-standard-8 \
 --vm-only \
 --tf-version=1.15.2 \
 --name=deeplab-tutorial
```
Note: If you have more than one project, you must specify the project name with the --project flag.
The configuration you specified appears. Enter y to approve or n to cancel.
When the ctpu up command has finished executing, verify that your shell prompt has changed from username@projectname to username@vm-name. This change shows that you are now logged into your Compute Engine VM.

Note: If you are not connected to the Compute Engine instance, you can connect by running the following commands:
```
gcloud compute ssh deeplab-tutorial --zone=us-central1-b
```

As you continue these instructions, run each command that begins with (vm)$ in your VM session window.

Install additional packages

For this model, you need to install the following additional packages on your Compute Engine instance:

jupyter
matplotlib
PrettyTable

  (vm)$ pip3 install --user jupyter

  (vm)$ pip3 install --user matplotlib

  (vm)$ pip3 install --user PrettyTable

Create environment variables for your storage bucket and TPU name.

(vm)$ export STORAGE_BUCKET=gs://bucket-name

(vm)$ export TPU_NAME=deeplab-tutorial
(vm)$ export DATA_DIR=${STORAGE_BUCKET}/deeplab_data
(vm)$ export MODEL_DIR=${STORAGE_BUCKET}/deeplab_model
(vm)$ export PYTHONPATH=${PYTHONPATH}:/usr/share/models/research:/usr/share/models/research/slim

Prepare the data set

Download and convert the PASCAL VOC 2012 dataset

This model uses the PASCAL VOC 2012 dataset for training and evaluation. Run the following script to download the dataset and convert it to TensorFlow's TFRecord format:
```
 (vm)$ bash /usr/share/models/research/deeplab/datasets/download_and_convert_voc2012.sh
```
Download the pretrained checkpoint

In this step, you download the modified resnet 101 pretrained checkpoint. To start, download the checkpoint:
```
 (vm)$ wget http://download.tensorflow.org/models/resnet_v1_101_2018_05_04.tar.gz
```
Then, extract the contents of the tar file:
```
 (vm)$ tar -vxf resnet_v1_101_2018_05_04.tar.gz
```
Upload data to your Cloud Storage bucket

At this point, you can now upload your data to the Cloud Storage bucket you created earlier:
```
(vm)$ gsutil -m cp -r pascal_voc_seg/tfrecord ${DATA_DIR}/tfrecord
```
```
(vm)$ gsutil -m cp -r resnet_v1_101 ${DATA_DIR}
```

Create Cloud TPU resource

Run the following command to create your Cloud TPU.

  (vm)$ ctpu up --tpu-only \
  --tf-version=1.15.2 \
  --tpu-size=v3-8 \
  --name=deeplab-tutorial

Train the model

Run the training script for 2000 training steps. This will take approximately 20 minutes. To run to convergence, remove the --train_steps=2000 flag from the training script command line. Running to convergence takes about 10 hours.

(vm)$ python3 /usr/share/tpu/models/experimental/deeplab/main.py \
--mode='train' \
--num_shards=8 \
--alsologtostderr=true \
--model_dir=${MODEL_DIR} \
--dataset_dir=${DATA_DIR}/tfrecord \
--init_checkpoint=${DATA_DIR}/resnet_v1_101/model.ckpt \
--model_variant=resnet_v1_101_beta \
--image_pyramid=1. \
--aspp_with_separable_conv=false \
--multi_grid=1 \
--multi_grid=2 \
--multi_grid=4 \
--decoder_use_separable_conv=false \
--train_split='train' \
--train_steps=2000 \
--tpu=${TPU_NAME}

Evaluating the model on a Cloud TPU device.

When the training completes, you can evaluate the model. To do so, change the --mode flag from train to eval:

(vm)$ python3 /usr/share/tpu/models/experimental/deeplab/main.py \
--mode='eval' \
--num_shards=8 \
--alsologtostderr=true \
--model_dir=${MODEL_DIR} \
--dataset_dir=${DATA_DIR}/tfrecord \
--init_checkpoint=${DATA_DIR}/resnet_v1_101/model.ckpt \
--model_variant=resnet_v1_101_beta \
--image_pyramid=1. \
--aspp_with_separable_conv=false \
--multi_grid=1 \
--multi_grid=2 \
--multi_grid=4 \
--decoder_use_separable_conv=false \
--train_split='train' \
--tpu=${TPU_NAME}

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Disconnect from the Compute Engine VM:
```
(vm)$ exit
```
Your prompt should now be username@projectname, showing you are in the Cloud Shell.
In your Cloud Shell, run ctpu delete with the --zone flag you used when you set up the Cloud TPU to delete your Compute Engine VM and your Cloud TPU:
```
$ ctpu delete --zone=us-central1-b \
  --name=deeplab-tutorial
```
Run ctpu status to make sure you have no instances allocated to avoid unnecessary charges for TPU usage. The deletion might take several minutes. A response like the one below indicates there are no more allocated instances:
```
2018/04/28 16:16:23 WARNING: Setting zone to "us-central1-b"
No instances currently exist.
        Compute Engine VM:     --
        Cloud TPU:             --
```
Run gsutil as shown, replacing bucket-name with the name of the Cloud Storage bucket you created for this tutorial:
```
$ gsutil rm -r gs://bucket-name
```

What's next

Learn more about ctpu, including how to install it on a local machine.
Experiment with more TPU samples.
Explore the TPU tools in TensorBoard.