Update a Dataproc cluster by using a template

Update a Dataproc cluster by using a template

This page shows you how to use an Google APIs Explorer template to update a Dataproc cluster to change the number of workers in a cluster. Scaling a cluster up to include more workers is a common task when additional workers are needed to process larger jobs.

For other ways to update a Dataproc cluster, see:

• Create a Dataproc cluster by using the Google Cloud console
• Create a Dataproc cluster by using the Google Cloud CLI
• Create a Dataproc cluster by using client libraries

Before you begin

This quickstart assumes you have already created a Dataproc cluster. You can use the APIs Explorer, the Google Cloud console, the gcloud CLI gcloud command-line tool, or the Quickstarts using Cloud Client Libraries to create a cluster.

Update a cluster

To update the number of workers in your cluster, fill in and execute the Google APIs Explorer Try this API template.

Note: The region, clusterName and updateMask and config.workerConfig.numInstances parameter values are filled in for you. Confirm or replace the region andclusterName parameter values to match your cluster's region and name. The updateMask parameter value is required to update the number of workers in your cluster. You can accept or change the config.workerConfig.numInstances parameter value.

1. Request parameters:

   1. Insert your projectId.
   2. Specify the region where your cluster is located (confirm or replace "us-central1"). Your cluster's region is listed on the Dataproc Clusters page in the Google Cloud console.
   3. Specify the clusterName of the existing cluster that you are updating (confirm or replace "example-cluster").
   4. updateMask: "config.worker_config.num_instances". This is the JSON PATH relative to the Cluster resource to the numInstances parameter to be updated (see the Request body instructions).

2. Request body:

   1. config.workerConfig.numInstances: ("3": the new number of workers). You can change this value to add fewer or more workers. For example, if your standard cluster has the default number of 2 workers, specifying "3" will add 1 worker; specifying "4 will add 2). A standard Dataproc cluster must have at least 2 workers.

3. Click EXECUTE. The first time you run the API template, you may be asked to choose and sign into your Google account, then authorize the Google APIs Explorer to access your account. If the request is successful, the JSON response shows that cluster update is pending.

4. To confirm that the number of workers in the cluster has been updated, open the Dataproc Clusters page in the Google Cloud console and view the cluster's Total worker nodes column. You may need to click REFRESH at the top of the page to view the updated value after the cluster update completes.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

1. If you don't need the cluster to explore the other quickstarts or to run other jobs, use the APIs Explorer, the Google Cloud console, the gcloud CLI gcloud command-line tool, or the Quickstarts using Cloud Client Libraries to delete the cluster.

What's next

• You can use this quickstart template to restore the cluster to its previous workerConfig.numInstances value.

• Learn how to write and run a Spark Scala job.