Download, pre-process, and upload the ImageNet dataset
This topic describes how to download, pre-process, and upload the ImageNet dataset to use with Cloud TPU VM architecture.
ImageNet is an image database. The images in the database are organized into a hierarchy, with each node of the hierarchy depicted by hundreds and thousands of images.The size of the ImageNet database means it can take a considerable amount of time to train a model. An alternative is to use a demonstration version of the dataset, referred to as fake_imagenet. This demonstration version lets you test the model, while reducing the storage and time requirements associated with using the full ImageNet database.
Pre-processing the full ImageNet dataset
The ImageNet dataset consists of three parts, training data, validation data, and image labels.
The training data contains 1000 categories and 1.2 million images, packaged for easy downloading. The validation and test data are not contained in the ImageNet training data (duplicates have been removed).
The validation and test data consists of 150,000 photographs, collected from Flickr and other search engines, hand labeled with the presence or absence of 1000 object categories. The 1000 object categories contain both internal nodes and leaf nodes of ImageNet, but do not overlap with each other. A random subset of 50,000 of the images with labels has been released as validation data along with a list of the 1000 categories. The remaining images are used for evaluation and have been released without labels.
Steps to pre-processing the full ImageNet dataset
There are five steps to preparing the full ImageNet dataset for use by a Machine Learning model:
- Verify that you have space on the download target.
- Set up the target directories.
- Register on the ImageNet site and request download permission.
Download the dataset to local disk or VM instance.
Run the pre-processing and upload script.
Verify space requirements
Whether you download the dataset to your local machine or to a VM instance, you need about 300 GB of space available on the download target.
The default disk allocation for a TPU VM is 100 GB. Since the download to your TPU VM requires
300 GB, if you are going download to your TPU VM instance, you will need
to add a persistent disk and with 200 GB of additional space to complete the download. On a TPU VM, you
can check your available storage with the df -ha
command.
When adding a persistent disk be sure to:
- Set When deleting instance to Delete disk to ensure that the disk is deleted when you delete the VM.
- Make a note of the path to your new disk. For example:
/mnt/disks/mnt-dir
.
Set up the target directories
On your local machine or VM instance, set up the directory structure to store the downloaded data.
Create and export a home directory for the ImageNet dataset.
Create a directory, for example,
imagenet
under your home directory on your download target (local machine or TPU VM). Under this directory, create two sub directories:train
andvalidation
. Export the home directory as IMAGENET_HOME:export IMAGENET_HOME=~/imagenet
Register and request permission to download the dataset
- Register on the Imagenet website. You cannot download the dataset until ImageNet confirms your registration and sends you a confirmation email. If you do not get the confirmation email within a couple of days, contact ImageNet support to see why your registration has not been confirmed. Once your registration is confirmed, you can download the dataset. The Cloud TPU tutorials that use the ImageNet dataset use the images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012).
Download the ImageNet dataset
From the LSRVC 2012 download site, go to the Images section on the page and right-click "Training images (Task 1 & 2)". The URL to download the largest part of the training set. Save the URL.
Right-click "Training images (Task 3)" to get the URL for the second training set. Save the URL.
Right-click "Validation images (all tasks)" to get the URL for the validation dataset. Save the URL.
If you download the ImageNet files to your local machine, you need to copy the directories on your local machine to the corresponding
$IMAGENET_HOME
directory on your VM instance. Copying the ImageNet dataset from local host to your VM instance takes approximately 13 hours.Before copying the ImageNet dataset to your VM, you need to identify the name of your VM instance. To do that, run the following
gcloud describe
command and locate your VM instance name in the output.gcloud compute tpus tpu-vm describe tpu-name --zone=zone
This generates output containing a line that includes your VM instance name (an example of a VM instance name is shown in bold below):
tpuVmSelflink: https://www.googleapis.com/compute/v1/projects/project-name/zones/zone/instances/t1v-n-2b9b54cd-w-0
Use the following command to copy the files under ~/imagenet on your local machine to
$IMAGENET_HOME
on your VM.gcloud compute scp --recurse $IMAGENET_HOME username@vm-instance-name:~/imagenet
From $IMAGENET_HOME, use
wget
to download the training and validation files using the saved URLs.The "Training images (Task 1 & 2)" file is the large training set. It is 138 GB and if you are downloading to your VM using the Cloud Shell, the download takes approximately 40 hours. If the Cloud Shell loses its connection to the VM, you can prepend
nohup
to the command or use screen.cd `$IMAGENET_HOME` \ nohup wget http://image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_train.tar
This command downloads a large tar file: ILSVRC2012_img_train.tar.
From
$IMAGENET_HOME
on the VM, extract the individual training directories into the$IMAGENET_HOME/train
directory using the following command. The extraction takes between 1 - 3 hours.tar xf $IMAGENET_HOME/ILSVRC2012_img_train.tar -C $IMAGENET_HOME/traintar xf ILSVRC2012_img_train.tar
Extract the individual training tar files located in the
$IMAGENET_HOME/train
directory, as shown in the following script:cd `$IMAGENET_HOME/train` for f in *.tar; do d=`basename $f .tar` mkdir $d tar xf $f -C $d done
Delete the tar files after you have extracted them to free up disk space.
The "Training images (Task 3)" file is 728 MB and takes just a few minutes to download so you do not need to take precautions against losing the Cloud Shell connection.
When you download this file, it extracts the individual training directories into the existing
$IMAGENET_HOME/train
directory.wget http://www.image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_train_t3.tar
When downloading the "Validation images (all tasks)" file, your Cloud Shell may disconnect. You can use
nohup
or screen to prevent Cloud Shell from disconnecting.wget http://www.image-net.org/challenges/LSVRC/2012/dd31405981ef5f776aa17412e1f0c112/ILSVRC2012_img_val.tar
This download takes about 30 minutes. When you download this file, it extracts the individual validation directories into the
$IMAGENET_HOME/validation
directory.If you downloaded the validation files to your local machine, you need to copy the
$IMAGENET_HOME/validation
directory on your local machine to the$IMAGENET_HOME/validation
directory on your VM instance. This copy operation takes about 30 minutes.Download the labels file.
wget -O $IMAGENET_HOME/synset_labels.txt \ https://raw.githubusercontent.com/tensorflow/models/master/research/slim/datasets/imagenet_2012_validation_synset_labels.txt
If you downloaded the labels file to your local machine, you need to copy it to the
$IMAGENET_HOME
directory on your local machine to$IMAGENET_HOME
on your VM instance. This copy operation takes a few seconds.The training subdirectory names (for example, n03062245) are "WordNet IDs" (wnid). The ImageNet API shows the mapping of WordNet IDs to their associated validation labels in the
synset_labels.txt
file. A synset in this context is a visually similar group of images.
Process the Imagenet dataset and, optionally, upload to Cloud Storage
Download the
imagenet_to_gcs.py
script from GitHub:wget https://raw.githubusercontent.com/tensorflow/tpu/master/tools/datasets/imagenet_to_gcs.py
If you are uploading the dataset to Cloud Storage, specify the storage bucket location to upload the ImageNet dataset:
export STORAGE_BUCKET=gs://bucket-name
If you are uploading the dataset to your local machine or VM, specify a data directory to hold the dataset:
(vm)$ export DATA_DIR=$IMAGENET_HOME/dataset-directory
Run the script to pre-process the raw dataset as TFRecords and upload it to Cloud Storage using the following command:
python3 imagenet_to_gcs.py \ --project=$PROJECT \ --gcs_output_path=$STORAGE_BUCKET \ --raw_data_dir=$IMAGENET_HOME \ --local_scratch_dir=$IMAGENET_HOME/tf_records
The script generates a set of directories (for both training and validation) of the form:
${DATA_DIR}/train-00000-of-01024
${DATA_DIR}/train-00001-of-01024
...
${DATA_DIR}/train-01023-of-01024
and
${DATA_DIR}/validation-00000-of-00128
S{DATA_DIR}/validation-00001-of-00128
...
${DATA_DIR}/validation-00127-of-00128
After the data has been uploaded to your Cloud bucket, run your model and set
--data_dir=${DATA_DIR}
.