Download, pre-process, and upload the ImageNet dataset

This topic describes how to download, pre-process, and upload the ImageNet dataset to use with Cloud TPU VM architecture.

ImageNet is an image database. The images in the database are organized into a hierarchy, with each node of the hierarchy depicted by hundreds and thousands of images.

The size of the ImageNet database means it can take a considerable amount of time to train a model. An alternative is to use a demonstration version of the dataset, referred to as fake_imagenet. This demonstration version lets you test the model, while reducing the storage and time requirements associated with using the full ImageNet database.

Pre-processing the full ImageNet dataset

The ImageNet dataset consists of three parts, training data, validation data, and image labels.

The training data contains 1000 categories and 1.2 million images, packaged for easy downloading. The validation and test data are not contained in the ImageNet training data (duplicates have been removed).

The validation and test data consists of 150,000 photographs, collected from Flickr and other search engines, hand labeled with the presence or absence of 1000 object categories. The 1000 object categories contain both internal nodes and leaf nodes of ImageNet, but do not overlap with each other. A random subset of 50,000 of the images with labels has been released as validation data along with a list of the 1000 categories. The remaining images are used for evaluation and have been released without labels.

Steps to pre-processing the full ImageNet dataset

There are five steps to preparing the full ImageNet dataset for use by a Machine Learning model:

Verify that you have space on the download target.
Set up the target directories.
Register on the ImageNet site and request download permission.
Download the dataset to local disk or VM instance.

Note: Downloading the Imagenet dataset to a VM instance takes considerably longer than downloading to your local machine (approximately 40 hours versus 7 hours). If you download the dataset to your local machine, you must copy the files to a VM instance to pre-process them. You must then upload the files to Cloud Storage before using them to train your model. Copying the training and validation files from your local machine to the VM instance takes about 13 hours. The recommended approach is to download the dataset to a VM instance.
Run the pre-processing and upload script.

Verify space requirements

Whether you download the dataset to your local machine or to a VM instance, you need about 300 GB of space available on the download target.

The default disk allocation for a TPU VM is 100 GB. Since the download to your TPU VM requires 300 GB, if you are going download to your TPU VM instance, you will need to add a persistent disk and with 200 GB of additional space to complete the download. On a TPU VM, you can check your available storage with the df -ha command.

When adding a persistent disk be sure to:

Set When deleting instance to Delete disk to ensure that the disk is deleted when you delete the VM.
Make a note of the path to your new disk. For example: /mnt/disks/mnt-dir.

Set up the target directories

On your local machine or VM instance, set up the directory structure to store the downloaded data.

Create and export a home directory for the ImageNet dataset.

Create a directory, for example, imagenet under your home directory on your download target (local machine or TPU VM). Under this directory, create two sub directories: train and validation. Export the home directory as IMAGENET_HOME:
```
export IMAGENET_HOME=~/imagenet
```

Register and request permission to download the dataset

Register on the Imagenet website. You cannot download the dataset until ImageNet confirms your registration and sends you a confirmation email. If you do not get the confirmation email within a couple of days, contact ImageNet support to see why your registration has not been confirmed. Once your registration is confirmed, you can download the dataset. The Cloud TPU tutorials that use the ImageNet dataset use the images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012).

Download the ImageNet dataset

From the LSRVC 2012 download site, go to the Images section on the page and right-click "Training images (Task 1 & 2)". The URL to download the largest part of the training set. Save the URL.

Right-click "Training images (Task 3)" to get the URL for the second training set. Save the URL.

Right-click "Validation images (all tasks)" to get the URL for the validation dataset. Save the URL.

If you download the ImageNet files to your local machine, you need to copy the directories on your local machine to the corresponding $IMAGENET_HOME directory on your VM instance. Copying the ImageNet dataset from local host to your VM instance takes approximately 13 hours.

Before copying the ImageNet dataset to your VM, you need to identify the name of your VM instance. To do that, run the following gcloud describe command and locate your VM instance name in the output.
```
gcloud compute tpus tpu-vm describe tpu-name --zone=zone
```
This generates output containing a line that includes your VM instance name (an example of a VM instance name is shown in bold below):

tpuVmSelflink: https://www.googleapis.com/compute/v1/projects/project-name/zones/zone/instances/t1v-n-2b9b54cd-w-0

Use the following command to copy the files under ~/imagenet on your local machine to $IMAGENET_HOME on your VM.
```
gcloud compute scp --recurse $IMAGENET_HOME username@vm-instance-name:~/imagenet
```
From $IMAGENET_HOME, use wget to download the training and validation files using the saved URLs.

The "Training images (Task 1 & 2)" file is the large training set. It is 138 GB and if you are downloading to your VM using the Cloud Shell, the download takes approximately 40 hours. If the Cloud Shell loses its connection to the VM, you can prepend nohup to the command or use screen.
```
cd `$IMAGENET_HOME` \
nohup wget http://image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_train.tar
```
This command downloads a large tar file: ILSVRC2012_img_train.tar.

From $IMAGENET_HOME on the VM, extract the individual training directories into the $IMAGENET_HOME/train directory using the following command. The extraction takes between 1 - 3 hours.
```
tar xf $IMAGENET_HOME/ILSVRC2012_img_train.tar -C $IMAGENET_HOME/traintar xf ILSVRC2012_img_train.tar
```
Extract the individual training tar files located in the $IMAGENET_HOME/train directory, as shown in the following script:
```
cd `$IMAGENET_HOME/train`

for f in *.tar; do
 d=`basename $f .tar`
 mkdir $d
 tar xf $f -C $d
done
```
Delete the tar files after you have extracted them to free up disk space.

The "Training images (Task 3)" file is 728 MB and takes just a few minutes to download so you do not need to take precautions against losing the Cloud Shell connection.

When you download this file, it extracts the individual training directories into the existing $IMAGENET_HOME/train directory.
```
wget http://www.image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_train_t3.tar
```
When downloading the "Validation images (all tasks)" file, your Cloud Shell may disconnect. You can use nohup or screen to prevent Cloud Shell from disconnecting.
```
wget http://www.image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_val.tar
```
This download takes about 30 minutes. When you download this file, it extracts the individual validation directories into the $IMAGENET_HOME/validation directory.

If you downloaded the validation files to your local machine, you need to copy the $IMAGENET_HOME/validation directory on your local machine to the $IMAGENET_HOME/validation directory on your VM instance. This copy operation takes about 30 minutes.

Download the labels file.
```
wget -O $IMAGENET_HOME/synset_labels.txt \
https://raw.githubusercontent.com/tensorflow/models/master/research/slim/datasets/imagenet_2012_validation_synset_labels.txt
```
If you downloaded the labels file to your local machine, you need to copy it to the $IMAGENET_HOME directory on your local machine to $IMAGENET_HOME on your VM instance. This copy operation takes a few seconds.

The training subdirectory names (for example, n03062245) are "WordNet IDs" (wnid). The ImageNet API shows the mapping of WordNet IDs to their associated validation labels in the synset_labels.txt file. A synset in this context is a visually similar group of images.

Process the Imagenet dataset and, optionally, upload to Cloud Storage

Download the imagenet_to_gcs.py script from GitHub:

wget https://raw.githubusercontent.com/tensorflow/tpu/master/tools/datasets/imagenet_to_gcs.py

If you are uploading the dataset to Cloud Storage, specify the storage bucket location to upload the ImageNet dataset:
```
export STORAGE_BUCKET=gs://bucket-name
```
If you are uploading the dataset to your local machine or VM, specify a data directory to hold the dataset:
```
(vm)$ export DATA_DIR=$IMAGENET_HOME/dataset-directory
```
Run the script to pre-process the raw dataset as TFRecords and upload it to Cloud Storage using the following command:

Note: If you don't want to upload to Cloud Storage, specify --nogcs_upload as another parameter and leave off the --project and --gcs_output_path parameters.
```
 python3 imagenet_to_gcs.py \
  --project=$PROJECT \
  --gcs_output_path=$STORAGE_BUCKET  \
  --raw_data_dir=$IMAGENET_HOME \
  --local_scratch_dir=$IMAGENET_HOME/tf_records
```

The script generates a set of directories (for both training and validation) of the form:

${DATA_DIR}/train-00000-of-01024
${DATA_DIR}/train-00001-of-01024
 ...
${DATA_DIR}/train-01023-of-01024

and

${DATA_DIR}/validation-00000-of-00128
S{DATA_DIR}/validation-00001-of-00128
 ...
${DATA_DIR}/validation-00127-of-00128

After the data has been uploaded to your Cloud bucket, run your model and set --data_dir=${DATA_DIR}.