Manage a cluster

This page explains how to update and delete an existing Dataproc cluster.

Update a cluster

You can update a cluster by issuing a Dataproc API clusters.patch request, running a gcloud dataproc clusters update command in a local terminal window or in Cloud Shell, or by editing cluster parameters from the Configuration tab of the Cluster details page for the cluster in the Google Cloud console.

The following cluster parameters can be updated:

• the number of standard worker nodes in a cluster—see Scaling clusters
• the number of secondary worker nodes in a cluster— see Preemptible VMs
• whether to use graceful decommissioning to control shutting down a worker after its jobs are completed
• adding or deleting cluster labels

Cluster update error

Dataproc issues an ERROR_DUE_TO_UPDATE error when a cluster update operation fails. This error places the cluster in a fail-safe state to avoid data loss, and restricts further cluster operations: jobs can continue to be submitted to the cluster, the cluster can be deleted, but further cluster update operations are not permitted.

Recommendation: After receiving an ERROR_DUE_TO_UPDATE on a cluster, recreate the cluster by exporting the cluster configuration to create a new cluster, then delete the failed cluster. If recreating the cluster fails, contact Cloud Customer Care for help in restoring the cluster to a RUNNING state.

Note: A cluster can enter the ERROR_DUE_TO_UPDATE state if you attempt to update cluster resources through Compute Engine API operations. To update cluster resources, use Dataproc update operations.

Delete a cluster

You can delete a cluster via a Dataproc API clusters.delete HTTP or programmatic request, using the Google Cloud CLI gcloud command-line tool locally in a terminal window or in Cloud Shell, or from the Google Cloud console.

Deletion is permanent: Once deleted, clusters cannot be restored. You can, however, quickly create new Dataproc clusters as you need them.

gcloud command

To delete a Dataproc cluster, use the gcloud CLI gcloud dataproc clusters delete command locally in a terminal window or in Cloud Shell.

gcloud dataproc clusters delete cluster-name \
    --region=region

REST API

Use the Dataproc clusters.delete API to delete a cluster.

Console

Open the Dataproc Clusters page in the Google Cloud console. Select the cluster by checking the box to the left of the cluster name, then Click Delete to delete the cluster.