Recreate and update a cluster

Dataproc image vulnerability:

Apache Log4j security vulnerabilities affect Dataproc images with subminor image version numbers prior to 1.4.79, 1.5.55, or 2.0.29. If your existing cluster was created with an affected image, Dataproc strongly recommends that you recreate existing clusters with an unaffected image. See the 2.0 image version, 1.5 image version, and 1.4 image version pages to select an image).

Unsupported 1.3 images: Subminor image versions prior to 1.3.95-debian10/-ubuntu18 are vulnerable. Dataproc has released a 1.3.95-debian10/-ubuntu18 image with a one-time patch that addresses the vulnerability, but note that that all 1.3 images remain unsupported, and Dataproc will not provide upgrades to 1.3 images .

New clusters: See Create a cluster for instructions to create a new cluster.

Custom images: Custom images built with an affected image version should be rebuilt with an unaffected image (see Create a Dataproc custom image).

Steps to recreate and update a cluster

You can use the gcloud command-line tool or the Dataproc API to copy configuration from an existing cluster, update the copied configuration, and then create a new cluster using the updated configuration.

gcloud command

The example instructions show updating the image version (NEW_IMAGE_VERSION) setting in a cluster configuration. The recommended practice is to specify the major.minor image version for production environments or when compatibility with specific component versions is important. The sub-minor and OS distributions will be automatically set to the latest weekly release.

You can change the name and use of this variable to update a different cluster configuration setting.

1. Set variables.

   export PROJECT=project-id
   export REGION=region
   export CLUSTER=cluster-name
   export NEW_IMAGE_VERSION=image-version (for example, '2.0-debian10')
   

2. Export the cluster configuration to a YAML file.

   gcloud dataproc clusters export $CLUSTER \
       --project=$PROJECT \
       --region=$REGION > "${CLUSTER}-config.yaml"

3. Update the configuration. The following example uses `sed` to update the image version.

   sed -E "s|(^[[:blank:]]+)imageVersion: .+|\1imageVersion: ${NEW_IMAGE_VERSION}|g" "${CLUSTER}-config.yaml" | sed -E '/^[[:blank:]]+imageUri: /d' > "${CLUSTER}-config-updated.yaml"

4. Delete the existing cluster. IMPORTANT: This step deletes all data stored in HDFS and on local disk in your cluster.

   gcloud dataproc clusters delete $CLUSTER \
       --project=$PROJECT \
       --region=$REGION

5. Import the updated cluster configuration to create a new cluster with the previous cluster's name and the updated settings.

   gcloud dataproc clusters import $CLUSTER \
       --project=$PROJECT \
       --region=$REGION \
       --source="${CLUSTER}-config-updated.yaml"

REST API

The example instructions show updating the image version (NEW_IMAGE_VERSION) setting in a cluster configuration. The recommended practice is to specify the major.minor image version for production environments or when compatibility with specific component versions is important. The sub-minor and OS distributions will be automatically set to the latest weekly release.

You can change the name and use of this variable to update a different cluster configuration setting.

1. Set variables.

   export CLUSTER=cluster-name
   export REGION=region
   export NEW_IMAGE_VERSION=image-version (for example, '2.0-debian10')
   

2. Export the cluster configuration to a JSON file.

   curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)"  "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}?alt=json" > "${CLUSTER}-config.json"

3. Update the configuration. The following example uses `jq` to update the image version.

   jq ".config.softwareConfig.imageVersion=\"${NEW_IMAGE_VERSION}\" | del(.config.workerConfig.imageUri) | del(.config.masterConfig.imageUri)" "${CLUSTER}-config.json" > "${CLUSTER}-config-updated.json"

5. Delete the existing cluster. IMPORTANT: This step deletes all data stored in HDFS and on local disk in your cluster.

   curl -X DELETE -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters/${CLUSTER}"
   

6. Wait for the previous operation returned by the call to finish, and then import the updated cluster configuration to create a new cluster with the previous cluster's name and the updated settings.

   curl -i -X POST  -H "Authorization: Bearer $(gcloud auth print-access-token)"  -H "Content-Type: application/json; charset=utf-8" -d "@${CLUSTER}-config-updated.json" "https://dataproc.googleapis.com/v1/projects/${PROJECT}/regions/${REGION}/clusters?alt=json"

Console

The console does not support recreating a cluster by importing a cluster's configuration.