Cloud Dataproc Versioning

Cloud Dataproc uses images to tie together useful Google Cloud Platform connectors and Apache Spark & Apache Hadoop components into one package that can be deployed on a Cloud Dataproc cluster. These images contain the base operating system (Debian) for the cluster, along with components needed to run jobs, such as Spark, Hadoop, Hive, and so on. These images will be upgraded periodically to include new improvements and features. Cloud Dataproc versioning allows you to select sets of software versions when you create clusters.

How versioning works

When an image is created, it is given an Image Version number in the following format:

version_major.version_minor.version_sub-minor

The recommended practice is to specify the major.minor image version for production environments or when compatibility with specific component versions is important.

Selecting versions

When you create a new Cloud Dataproc cluster, the latest available image version will be used by default. You can select an image version when creating a new cluster.

gcloud Command

When using the gcloud dataproc clusters create command, you can use the --image-version argument to specify an image version. For example, you can run the following command to create a new my-test-cluster that uses the current sub-minor release of image version 1.0.

gcloud dataproc clusters create my-test-cluster --image-version 1.0

Best practice is to specify the major and minor version only, so that the latest sub-minor version is always used. However, if necessary, the sub-minor version can be specified as well.

You can check your current version with the gcloud command-line tool.

gcloud dataproc clusters describe cluster-name

REST API

You can specify the SoftwareConfig imageVersion field as part of a cluster.create API request.

Example

POST /v1/projects/project-id/regions/global/clusters/
{
  "projectId": "project-id",
  "clusterName": "example-cluster",
  "config": {
    "configBucket": "",
    "gceClusterConfig": {
      "subnetworkUri": "default",
      "zoneUri": "us-central1-b"
    },
    "masterConfig": {
      ...
      }
    },
    "workerConfig": {
      ...
      }
    },
    "softwareConfig": {
      "imageVersion": "1.2"
    }
  }
}
  

Console

When creating a new cluster, click Advanced options at bottom of the Cloud Dataproc Create a cluster form.

The Image field shows the image that will be used when creating the cluster. Initially, it shows the default (latest available version).

Click Change to display a lists of available images you can select to use for your cluster. You select a standard or custom image.

When new versions are created

New major versions will be created periodically to incorporate one or more of the following:

  • Major releases for:
    • Operating system
    • Spark, Hadoop, and other Big Data components
    • Google Cloud connectors
  • Major changes or updates to Cloud Dataproc functionality

New minor versions will be created periodically to incorporate one or more of the following:

  • Minor releases and updates for:
    • Operating system
    • Spark, Hadoop, and other Big Data components
    • Google Cloud connectors
  • Minor changes or updates to Cloud Dataproc functionality

When a new minor version is created, it becomes the default for the major version, and represents the latest release of the major version.

New sub-minor versions will be created periodically to incorporate one or more of the following:

  • Patches or fixes for a component in the image

Image Version and Cloud Dataproc support

Major and minor image versions are supported for a specified period of time after they are released. During this period, clusters using the Image Versions are eligible for support. After the support window has closed, clusters using the Image Version are not eligible for support.

Months after Image Version release Can create clusters with this Image Version? Clusters using this Image Version eligible for support?
0-12 Yes Yes
12-24 Yes No
24+ No No

Sub-minor versions do not have guaranteed lifetimes or support.

Important notes about versioning

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataproc Documentation