Cloud ML Versioning

Google Cloud Machine Learning uses images to configure the VMs that service your training and prediction requests in the cloud. These images contain the base operating system (Debian), core technology packages, pip packages (Python libraries), and operating system packages. Images are upgraded periodically to include new improvements and features. Cloud ML versioning enables you to select the right configuration to work with your model.

Understanding version numbers

The images that Cloud ML uses correspond to the Cloud ML runtime version. The runtime version uses the following format:

major_version.minor_version

Major versions

New major versions are created periodically to incorporate one or more of the following:

  • Major releases for:
    • Operating system
    • TensorFlow
  • Major changes or updates to Cloud ML functionality.

A new major version may include breaking changes that require updates to code written against previous versions.

Minor versions

New minor versions are created periodically to incorporate one or more of the following:

  • Minor releases and updates for:
    • Operating system
    • TensorFlow
  • Minor changes or updates to Cloud ML functionality.

A new minor version should not include breaking changes, and should be backward-compatible with all variations of the same major version.

Supported versions

You'll find the details of each version in the Cloud ML version list.

Selecting runtime versions

The version you select is a function of the version of the key packages you use to develop your training application. In practice the TensorFlow version is typically the most important factor. We recommend that you develop your trainer with the most recent supported versions of all packages.

Specifying a runtime version is optional. Cloud ML uses its default version if you don't specify one.

To specify the runtime version for a training job

You can set the runtime version when you submit a training job request.

command line

Use the --runtime-version flag when you run the gcloud ml-engine jobs submit training command:

gcloud ml-engine jobs submit training my_job \
    --module-name trainer.task \
    --job-dir gs://my/training/job/directory \
    --package-path /path/to/my/project/trainer \
    --region us-central1
    --runtime-version 0.12

python

Set the optional runtimeVersion member of the TrainingInput object to the desired runtime version:

training_inputs = {'scaleTier': 'BASIC',
    'packageUris': ['gs://my/trainer/path/package-0.0.0.tar.gz'],
    'pythonModule': 'trainer.task'
    'args': ['--arg1', 'value1', '--arg2', 'value2'],
    'region': 'us-central1',
    'jobDir': 'gs://my/training/job/directory',
    'runtimeVersion': '0.12'}

job_spec = {'jobId': my_job_name, 'trainingInput': training_inputs}

To specify the runtime version for a model version

You can specify a runtime version when you create a deployed model version from a trained model. If you do, that version is used by default when you request online or batch predictions. If you don't, your model version's default runtime version is set to Cloud ML's current default runtime version.

command line

Use the --runtime-version flag when you run the gcloud ml-engine versions create command:

gcloud ml-engine models versions create version_name \
    --model model_name
    --origin gs://my/trained/model/path
    --runtime-version 0.12

Python

Set the optional runtimeVersion member of the Version resource to the desired runtime version:

versionDef = {'name' = 'v1',
    'description' = 'The first iteration of the completely_made_up model',
    'deploymentUri' = 'gs://my/model/output/directory',
    'runtimeVersion' = '0.12'}
 

To specify a runtime version to use for batch prediction

You can specify a runtime version to use when you create a batch prediction job. If you don't, Cloud ML uses the default runtime version set in the model version.

command line

Use the --runtime-version flag when you run the gcloud ml-engine jobs submit prediction command:

gcloud ml-engine jobs submit prediction my_batch_job_333 \
    --model my_model \
    --input-paths gs://my/cloud/storage/data/path/* \
    --output-path gs://my/cloud/storage/data/output/path
    --region us-central1 \
    --data-format TEXT \
    --runtime-version 0.12

Python

Set the optional runtimeVersion member of the PredictionInput object to the desired runtime version:

body = {
    'jobId': 'my_batch_job_333',
    'predictionInput': {
        'dataFormat': TEXT,
        'inputPaths': ['gs://my/cloud/storage/data/path/*'],
        'outputPath': 'gs://my/cloud/storage/data/output/path',
        'region': 'us-central1',
        'modelName': 'projects/my_project/models/my_model',
        'runtimeVersion': '0.12'}}

Runtime versions for online prediction

You cannot specify a runtime version when requesting online prediction. It always uses the model version's default runtime version.

Specifying variant packages (training only)

There are two ways for your to change the packages on your training instances: manually uploading package files (tarballs) and including their paths as training input, and specifying PyPI packages as dependencies to your trainer package.

To provide package files

You can include extra package files as part of your training job request. These will be installed on each training instance. Cloud ML installs all packages with pip. Packages designed for other package managers are not supported.

command line

Use the --packages flag when you run the gcloud ml-engine jobs submit training command. Set the value to a comma-separated list of the paths to all additional packages. Note that the list can contain no whitespace between entries.

gcloud ml jobs submit training my_job \
    --staging-bucket gs://my-bucket \
    --package-path /path/to/my/project/trainer \
    --module-name trainer.task \
    --packages dep1.tar.gz,dep2.whl

python

Add all additional packages to the list you use for the value of packageUris in the TrainingInput object.

training_inputs = {'scaleTier': 'BASIC',
    'packageUris': ['gs://my/trainer/path/package-0.0.0.tar.gz',
                    'gs://my/dependencies/path/dep1.tar.gz',
                    'gs://my/dependencies/path/dep2.whl'],
    'pythonModule': 'trainer.task'
    'args': ['--arg1', 'value1', '--arg2', 'value2'],
    'region': 'us-central1',
    'jobDir': 'gs://my/training/job/directory',
    'runtimeVersion': '0.12'}

job_spec = {'jobId': my_job_name, 'trainingInput’: training_inputs}

To include PyPI package dependencies

You can specify PyPI packages and their versions as dependencies to your trainer package using the normal setup tools process:

  1. In the top-level directory of your trainer application, include a setup.py file.
  2. When you call setuptools.setup in setup.py, pass a list of dependencies and optionally their versions as the install_requires parameter. This example setup.py file demonstrates the procedure:

    from setuptools import find_packages
    from setuptools import setup

    REQUIRED_PACKAGES = ['some_PyPI_package>=1.5', 'another_package==2.6']

    setup( name='trainer', version='0.1', install_requires=REQUIRED_PACKAGES, include_package_data=True, description='Generic example trainer package with dependencies.')

Cloud ML forces reinstallation of packages, so you can override packages that are part of the runtime version's image with newer or older versions.

Runtime version and Cloud ML support

Major and minor runtime versions are supported for a specific period of time after the are released. You can continue to train models and get predictions using a runtime version during this period. Runtime versions are not supported beyond that window.

Months after runtime is superseded Can train models and serve predictions
0-12 Yes
12+ No

Important notes about versioning

  • You should always test your training jobs and models thoroughly when switching to a new runtime version, regardless of whether it's a major or minor update.

What's next

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Cloud Machine Learning Engine (Cloud ML Engine)