Download, pre-process, and upload the ImageNet dataset

This document describes how to download, pre-process, and upload the ImageNet dataset to use with Cloud TPU VM architecture.

ImageNet is an image database. The images in the database are organized into a hierarchy, with each node of the hierarchy depicted by hundreds and thousands of images.

The size of the ImageNet database means it can take a considerable amount of time to train a model. An alternative is to use a demonstration version of the dataset, referred to as fake_imagenet. This demonstration version lets you test the model, while reducing the storage and time requirements associated with using the full ImageNet database.

Pre-processing the full ImageNet dataset

The ImageNet dataset consists of three parts, training data, validation data, and image labels.

The training data contains 1000 categories and 1.2 million images, packaged for downloading. The validation and test data are not contained in the ImageNet training data (duplicates have been removed).

The validation and test data consists of 150,000 photographs, collected from Flickr and other search engines, hand labeled with the presence or absence of 1000 object categories. The 1000 object categories contain both internal nodes and leaf nodes of ImageNet, but don't overlap with each other. A random subset of 50,000 of the images with labels has been released as validation data along with a list of the 1000 categories. The remaining images are used for evaluation and have been released without labels.

Steps to pre-process the full ImageNet dataset

There are five steps to preparing the full ImageNet dataset for use by a machine learning model:

Verify that you have space on the download target.
Set up the target directories.
Register on the ImageNet site and request download permission.
Download the dataset to local disk or VM instance.

Note: Downloading the Imagenet dataset to a VM instance takes considerably longer than downloading to your local machine (approximately 40 hours versus 7 hours). If you download the dataset to your local machine, you must copy the files to a VM instance to pre-process them. You must then upload the files to Cloud Storage before using them to train your model. Copying the training and validation files from your local machine to the VM instance takes about 13 hours. The recommended approach is to download the dataset to a VM instance.
Run the pre-processing and upload script.

Verify space requirements

Whether you download the dataset to your local machine or to a VM instance, you need about 300 GB of space available on the download target.

The default disk allocation for a TPU VM is 100 GB. Since the download to your TPU VM requires 300 GB, if you are going download to your TPU VM instance, you will need to add a persistent disk and with 200 GB of additional space to complete the download. On a TPU VM, you can check your available storage with the df -ha command.

When adding a persistent disk be sure to:

Set When deleting instance to Delete disk to ensure that the disk is deleted when you delete the VM.
Make a note of the path to your new disk. For example: /mnt/disks/mnt-dir.

Set up the target directories

On your local machine or VM instance, set up the directory structure to store the downloaded data.

Create a directory, for example, imagenet, under your home directory on your download target (local machine or TPU VM).
```
mkdir imagenet
```
Under this directory, create two sub directories: train and validation.
```
mkdir imagenet/train
mkdir imagenet/validation
```
Export the imagenet directory as IMAGENET_HOME.
```
export IMAGENET_HOME=~/imagenet
```

Register and request permission to download the dataset

Register on the Imagenet website. You cannot download the dataset until ImageNet confirms your registration and sends you a confirmation email. If you don't get the confirmation email within a couple of days, contact ImageNet support to see why your registration has not been confirmed. Once your registration is confirmed, you can download the dataset. The Cloud TPU tutorials that use the ImageNet dataset use the images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012).

Download the ImageNet dataset

From the LSRVC 2012 download site, go to the Images section on the page and right-click "Training images (Task 1 & 2)". The URL to download the largest part of the training set. Save the URL.

Right-click "Training images (Task 3)" to get the URL for the second training set. Save the URL.

Right-click "Validation images (all tasks)" to get the URL for the validation dataset. Save the URL.

If you download the ImageNet files to your local machine, you need to copy the directories on your local machine to the corresponding $IMAGENET_HOME directory on your VM instance. Copying the ImageNet dataset from local host to your VM instance takes approximately 13 hours.

Before copying the ImageNet dataset to your TPU VM, you need to identify the name of your TPU VM instance. To do that, connect to your TPU VM using SSH. The VM instance name is shown in the command prompt following the @ symbol.

Use the following command to copy the files under ~/imagenet on your local machine to $IMAGENET_HOME on your VM.
```
gcloud compute scp --recurse $IMAGENET_HOME username@vm-instance-name:~/imagenet
```
From $IMAGENET_HOME, download the "Training images (Task 1 & 2)" files using the saved URL.

The "Training images (Task 1 & 2)" file is the large training set. It is 138 GB, and if you are downloading to your TPU VM using the Cloud Shell, the download takes approximately 40 hours. If the Cloud Shell loses its connection to the VM, you can prepend nohup to the command or use screen.
```
cd $IMAGENET_HOME \
nohup wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_train.tar
```
Replace the URL with the URL you previously saved for "Training images (Task 1 & 2)", as the URL might have changed.
Extract the individual training directories into the $IMAGENET_HOME/train directory using the following command. The extraction takes between 1 - 3 hours.
```
tar xf $IMAGENET_HOME/ILSVRC2012_img_train.tar -C $IMAGENET_HOME/train
```
Extract the individual training tar files located in the $IMAGENET_HOME/train directory, as shown in the following script:
```
cd `$IMAGENET_HOME/train`

for f in *.tar; do
 d=`basename $f .tar`
 mkdir $d
 tar xf $f -C $d
done
```
Delete the tar files after you have extracted them to free up disk space.
From $IMAGENET_HOME, download the "Training images (Task 3)" files using the saved URL.

The "Training images (Task 3)" file is 728 MB and takes just a few minutes to download, so you don't need to take precautions against losing the Cloud Shell connection.
```
wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_train_t3.tar
```
Replace the URL with the URL you previously saved for "Training images (Task 3)", as the URL might have changed.
Extract the individual training directories into the $IMAGENET_HOME/train directory using the following command.
```
tar xf $IMAGENET_HOME/ILSVRC2012_img_train_t3.tar -C $IMAGENET_HOME/train
```
Extract the individual training tar files located in the $IMAGENET_HOME/train directory, as shown in the following script:
```
cd `$IMAGENET_HOME/train`

for f in *.tar; do
 d=`basename $f .tar`
 mkdir $d
 tar xf $f -C $d
done
```
Delete the tar files after you have extracted them to free up disk space.
From $IMAGENET_HOME, download the "Validation images (all tasks)" files using the saved URL.

When downloading the "Validation images (all tasks)" file, your Cloud Shell may disconnect. You can use nohup or screen to prevent Cloud Shell from disconnecting.
```
wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
```
Replace the URL with the URL you previously saved for "Validation images (all tasks)", as the URL might have changed.

This download takes about 30 minutes.
Extract the individual validation files into the $IMAGENET_HOME/validation directory using the following command.
```
tar xf $IMAGENET_HOME/ILSVRC2012_img_val.tar -C $IMAGENET_HOME/validation
```
If you downloaded the validation files to your local machine, you need to copy the $IMAGENET_HOME/validation directory on your local machine to the $IMAGENET_HOME/validation directory on your VM instance. This copy operation takes about 30 minutes.
Download the labels file.
```
wget -O $IMAGENET_HOME/synset_labels.txt \
https://raw.githubusercontent.com/tensorflow/models/master/research/slim/datasets/imagenet_2012_validation_synset_labels.txt
```
If you downloaded the labels file to your local machine, you need to copy it to the $IMAGENET_HOME directory on your local machine to $IMAGENET_HOME on your VM instance. This copy operation takes a few seconds.

The training subdirectory names (for example, n03062245) are "WordNet IDs" (wnid). The ImageNet API shows the mapping of WordNet IDs to their associated validation labels in the synset_labels.txt file. A synset in this context is a visually similar group of images.

Process the Imagenet dataset and, optionally, upload to Cloud Storage

Download the imagenet_to_gcs.py script from GitHub:

wget https://raw.githubusercontent.com/tensorflow/tpu/master/tools/datasets/imagenet_to_gcs.py

If you are uploading the dataset to Cloud Storage, specify the storage bucket location to upload the ImageNet dataset:
```
export STORAGE_BUCKET=gs://bucket-name
```
If you are uploading the dataset to your local machine or VM, specify a data directory to hold the dataset:
```
(vm)$ export DATA_DIR=$IMAGENET_HOME/dataset-directory
```
Install the following libraries on your machine or in a virtual environment:
```
pip3 install google-cloud-storage
pip3 install tensorflow
```
Run the script to pre-process the raw dataset as TFRecords and upload it to Cloud Storage using the following command:

Note: If you don't want to upload to Cloud Storage, specify --nogcs_upload as another parameter and leave off the --project and --gcs_output_path parameters.
```
 python3 imagenet_to_gcs.py \
  --project=$PROJECT \
  --gcs_output_path=$STORAGE_BUCKET  \
  --raw_data_dir=$IMAGENET_HOME \
  --local_scratch_dir=$IMAGENET_HOME/tf_records
```

The script generates a set of directories (for both training and validation) of the form:

${DATA_DIR}/train-00000-of-01024
${DATA_DIR}/train-00001-of-01024
...
${DATA_DIR}/train-01023-of-01024

and

${DATA_DIR}/validation-00000-of-00128
${DATA_DIR}/validation-00001-of-00128
...
${DATA_DIR}/validation-00127-of-00128

After the data has been uploaded to your Cloud bucket, run your model and set --data_dir=${DATA_DIR}.