Pre-built containers for custom training

AI Platform (Unified) provides Docker container images that you run as pre-built containers for custom training. These containers, which are organized by machine learning (ML) framework and framework version, include common dependencies that you might want to use in training code. In many cases, using a pre-built container is simpler than creating your own custom container for training.

This document lists the pre-built containers for training, and it describes how to use them with a Python training application.

Available container images

Each of the following container images is available in several Artifact Registry repositories, which store data in various locations. You can use any of the URIs for an image when you perform custom training; each provides the same container image. If you use the Google Cloud Console to perform custom training, the Cloud Console selects the URI that best matches the location where you are using AI Platform in order to reduce latency.

TensorFlow

ML framework version Use with GPUs? URIs (choose any) Included dependencies

2.4

us-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-4:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-4:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-4:latest

PyPI packages

Ubuntu packages

tensorflow 2.4.x (latest patch)
numpy 1.19.4
pandas 1.1.5
scipy 1.5.4
scikit-learn 0.24.0
sympy 1.7.1
statsmodels 0.12.1
xgboost 1.3.1
oauth2client 4.1.3
httplib2 0.15.0
python-dateutil 2.8.1
six 1.15.0
future 0.18.2
PyYAML 3.13
wrapt 1.12.1
crcmod 1.7
google-api-python-client 1.9.0
python-json-logger 2.0.1
wheel 0.36.2
WebOb 1.8.6
Paste 3.5.0
tornado 5.1.1
grpcio 1.32.0
requests 2.25.1
webapp2 3.0.0b1
mock 4.0.3
google-cloud-bigquery 2.6.1
google-cloud-bigtable 1.6.1
google-cloud-datastore 2.1.0
google-cloud-logging 1.15.0
google-cloud-pubsub 2.2.0
google-cloud-resource-manager 0.30.3
google-cloud-storage 1.35.0
joblib 1.0.0
cloudml-hypertune
psutil 5.8.0

curl
libcurl3-dev
wget
zip
unzip
git
vim
build-essential
ca-certificates
pkg-config
rsync
ca-certificates-java
libatlas-base-dev
liblapack-dev
gfortran
libpng-dev
libjpeg-dev
python3.7
python3-dev
python3.7-dev
python3-pip
python3-setuptools
python2.7
python-dev
python-setuptools
gdb
openjdk-8-jdk
openjdk-8-jre-headless
g++
zlib1g-dev
libio-all-perl
module-init-tools
libsnappy-dev
libyaml-0-2
python-opencv

2.4

Yes

us-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-4:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-4:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-4:latest

PyPI packages

Ubuntu packages

2.3

us-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-3:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-3:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-3:latest

PyPI packages

Ubuntu packages

tensorflow 2.3.x (latest patch)
numpy 1.18.5
pandas 1.1.3
scipy 1.4.1
scikit-learn 0.23.2
sympy 1.6.2
statsmodels 0.12.0
xgboost 1.2.1
oauth2client 4.1.3
httplib2 0.15.0
python-dateutil 2.8.1
six 1.15.0
future 0.18.2
PyYAML 3.13
wrapt 1.12.1
crcmod 1.7
google-api-python-client 1.9.0
python-json-logger 2.0.1
wheel 0.35.1
WebOb 1.8.6
Paste 3.5.0
tornado 5.1.1
grpcio 1.33.1
requests 2.24.0
webapp2 3.0.0b1
mock 4.0.2
google-cloud-bigquery 2.2.0
google-cloud-bigtable 1.5.1
google-cloud-datastore 1.15.3
google-cloud-logging 1.15.0
google-cloud-pubsub 2.1.0
google-cloud-resource-manager 0.30.2
google-cloud-storage 1.32.0
joblib 0.17.0
cloudml-hypertune
psutil 5.7.2

2.3

Yes

us-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-3:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-3:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-3:latest

PyPI packages

Ubuntu packages

2.2

us-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-2:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-2:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-2:latest

PyPI packages

Ubuntu packages

tensorflow 2.2.x (latest patch)
numpy 1.18.5
pandas 1.0.4
scipy 1.4.1
scikit-learn 0.23.1
sympy 1.6
statsmodels 0.11.1
xgboost 1.1.1
oauth2client 4.1.3
httplib2 0.18.1
python-dateutil 2.8.1
six 1.15.0
future 0.18.2
PyYAML 5.3.1
wrapt 1.12.1
crcmod 1.7
google-api-python-client 1.9.3
python-json-logger 0.1.11
wheel 0.34.2
WebOb 1.8.6
Paste 3.4.1
tornado 5.1.1
grpcio 1.29.0
requests 2.23.0
webapp2 3.0.0b1
mock 4.0.2
google-cloud-bigquery 1.25.0
google-cloud-bigtable 1.2.1
google-cloud-datastore 1.12.0
google-cloud-logging 1.15.0
google-cloud-pubsub 1.6.0
google-cloud-resource-manager 0.30.2
google-cloud-storage 1.29.0
joblib 0.15.1
cloudml-hypertune
psutil 5.7.0

2.2

Yes

us-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest

PyPI packages

Ubuntu packages

2.1

us-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-1:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-1:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.2-1:latest

PyPI packages

Ubuntu packages

tensorflow 2.1.0
numpy 1.18.0
pandas 0.25.3
scipy 1.4.1
scikit-learn 0.22.1
sympy 1.5
statsmodels 0.10.2
xgboost 0.90
oauth2client 4.1.3
httplib2 0.15.0
python-dateutil 2.8.1
six 1.13.0
future 0.18.2
PyYAML 5.2
wrapt 1.11.2
crcmod 1.7
google-api-python-client 1.7.11
python-json-logger 0.1.11
wheel 0.33.6
WebOb 1.8.5
Paste 3.2.3
tornado 5.1.1
grpcio 1.26.0
requests 2.22.0
webapp2 3.0.0b1
mock 3.0.5
google-cloud-bigquery 1.23.1
google-cloud-bigtable 1.2.0
google-cloud-datastore 1.10.0
google-cloud-logging 1.14.0
google-cloud-pubsub 1.1.0
google-cloud-resource-manager 0.30.0
google-cloud-storage 1.23.0
joblib 0.14.1
cloudml-hypertune
psutil 5.7.0

2.1

Yes

us-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.2-2:latest

PyPI packages

Ubuntu packages

1.15

us-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.1-15:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.1-15:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-cpu.1-15:latest

PyPI packages

Ubuntu packages

tensorflow 1.15.0
numpy 1.16.5
pandas 0.24.2
scipy 1.2.2
scikit-learn 0.20.4
sympy 1.4
statsmodels 0.10.1
xgboost 0.82
oauth2client 4.1.3
httplib2 0.12.0
python-dateutil 2.7.5
argparse 1.4.0
six 1.12.0
future 0.17.1
PyYAML 3.13
wrapt 1.11.1
crcmod 1.7
google-api-python-client 1.7.8
python-json-logger 0.1.10
wheel 0.32.3
WebOb 1.8.5
Paste 3.0.6
tornado 5.1.1
grpcio 1.18.0
requests 2.19.0
webapp2 3.0.0b1
mock 2.0.0
google-cloud-bigquery 1.20.0
google-cloud-bigtable 1.0.0
google-cloud-datastore 1.9.0
google-cloud-logging 1.12.1
google-cloud-pubsub 1.0.0
google-cloud-resource-manager 0.29.2
google-cloud-storage 1.19.1
joblib 0.13.0
cloudml-hypertune
psutil 5.7.0

1.15

Yes

us-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.1-15:latest
europe-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.1-15:latest
asia-docker.pkg.dev/cloud-aiplatform/training/tf-gpu.1-15:latest

PyPI packages

Ubuntu packages

scikit-learn

ML framework version Use with GPUs? URIs (choose any) Included dependencies

0.23

us-docker.pkg.dev/cloud-aiplatform/training/scikit-learn-cpu.0-23:latest
europe-docker.pkg.dev/cloud-aiplatform/training/scikit-learn-cpu.0-23:latest
asia-docker.pkg.dev/cloud-aiplatform/training/scikit-learn-cpu.0-23:latest

PyPI packages

Ubuntu packages

numpy 1.18.5
pandas 1.0.4
scipy 1.4.1
scikit-learn 0.23.1
sympy 1.6
statsmodels 0.11.1
xgboost 1.1.1
oauth2client 4.1.3
httplib2 0.18.1
python-dateutil 2.8.1
six 1.15.0
future 0.18.2
PyYAML 5.3.1
wrapt 1.12.1
crcmod 1.7
google-api-python-client 1.9.3
python-json-logger 0.1.11
wheel 0.34.2
WebOb 1.8.6
Paste 3.4.1
tornado 5.1.1
grpcio 1.29.0
requests 2.23.0
webapp2 3.0.0b1
mock 4.0.2
google-cloud-bigquery 1.25.0
google-cloud-bigtable 1.2.1
google-cloud-datastore 1.12.0
google-cloud-logging 1.15.0
google-cloud-pubsub 1.6.0
google-cloud-resource-manager 0.30.2
google-cloud-storage 1.29.0
joblib 0.15.1
cloudml-hypertune

PyTorch

ML framework version	Use with GPUs?	URIs (choose any)	Included dependencies
1.7	No	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-7:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-7:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-7:latest`	AI Platform Deep Learning Containers dependencies
1.7	Yes	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-7:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-7:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-7:latest`	AI Platform Deep Learning Containers dependencies
1.6	No	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-6:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-6:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-xla.1-6:latest`	AI Platform Deep Learning Containers dependencies
1.6	Yes	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-6:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-6:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-6:latest`	AI Platform Deep Learning Containers dependencies
1.4	No	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-cpu.1-4:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-cpu.1-4:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-cpu.1-4:latest`	AI Platform Deep Learning Containers dependencies
1.4	Yes	`us-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-4:latest` `europe-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-4:latest` `asia-docker.pkg.dev/cloud-aiplatform/training/pytorch-gpu.1-4:latest`	AI Platform Deep Learning Containers dependencies

XGBoost

ML framework version Use with GPUs? URIs (choose any) Included dependencies

1.1

us-docker.pkg.dev/cloud-aiplatform/training/xgboost-cpu.1-1:latest
europe-docker.pkg.dev/cloud-aiplatform/training/xgboost-cpu.1-1:latest
asia-docker.pkg.dev/cloud-aiplatform/training/xgboost-cpu.1-1:latest

PyPI packages

Ubuntu packages

Using a pre-built container

To use a pre-built container, read the guide to configuring container settings.

If you are using a container image that supports GPUs, make sure to specify the machineSpec.acceleratorType and machineSpec.acceleratorCount fields when you create your CustomJob, HyperparameterTuningJob, or TrainingPipeline resource.

What's next

Learn how to create a Python training application to use with a pre-built container.