Update a cluster

This sample walks a user through updating a Cloud Dataproc cluster using the Python client library.

Code sample

Python

Before trying this sample, follow the Python setup instructions in the Dataproc quickstart using client libraries. For more information, see the Dataproc Python API reference documentation.

To authenticate to Dataproc, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

from google.cloud import dataproc_v1 as dataproc


def update_cluster(project_id, region, cluster_name, new_num_instances):
    """This sample walks a user through updating a Cloud Dataproc cluster
    using the Python client library.

    Args:
        project_id (str): Project to use for creating resources.
        region (str): Region where the resources should live.
        cluster_name (str): Name to use for creating a cluster.
    """

    # Create a client with the endpoint set to the desired cluster region.
    client = dataproc.ClusterControllerClient(
        client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"}
    )

    # Get cluster you wish to update.
    cluster = client.get_cluster(
        project_id=project_id, region=region, cluster_name=cluster_name
    )

    # Update number of clusters
    mask = {"paths": {"config.worker_config.num_instances": str(new_num_instances)}}

    # Update cluster config
    cluster.config.worker_config.num_instances = new_num_instances

    # Update cluster
    operation = client.update_cluster(
        project_id=project_id,
        region=region,
        cluster=cluster,
        cluster_name=cluster_name,
        update_mask=mask,
    )

    # Output a success message.
    updated_cluster = operation.result()
    print(f"Cluster was updated successfully: {updated_cluster.cluster_name}")

What's next

To search and filter code samples for other Google Cloud products, see the Google Cloud sample browser.