Overview of Cloud Dataproc Workflow Templates

The Cloud Dataproc WorkflowTemplates API provides a flexible and easy-to-use mechanism for managing and executing workflows. A Workflow Template is a reusable workflow configuration. It defines a graph of jobs with information on where to run those jobs.

Key Points:

  • Instantiating a Workflow Template launches a Workflow. A Workflow is an operation that runs a Directed Acyclic Graph (DAG) of jobs on a cluster. If the workflow uses a managed cluster, it creates the cluster, runs the jobs, and then deletes the cluster when the jobs are finished.
  • If the workflow uses a cluster selector, it runs jobs on a selected existing cluster.
  • Workflows are ideal for complex job flows. You can create job dependencies so that so that a job starts only after its dependencies complete successfully.
  • Creating a workflow template does not create a Cloud Dataproc cluster or submit jobs. Clusters and jobs associated with workflows are created when a workflow template is instantiated.

Kinds of workflow templates

Managed cluster

A workflow template can specify a managed cluster. The workflow will create this "ephemeral" cluster to run workflow jobs, and then delete the cluster when the workflow is finished.

Cluster selector

A workflow template can specify an existing cluster on which to run workflow jobs by specifying one or more user labels that were previously applied to one or more clusters. The workflow will run on a cluster that matches all of the specified labels. If multiple clusters match the label(s), Cloud Dataproc will select the cluster with the most YARN available memory to run all workflow jobs. At the end of workflow, the selected cluster is not deleted. See Using cluster selectors with workflows for more information.

Parameterized

If you will run a workflow template multiple times with different values, you can avoid editing the workflow template for each run by defining parameters in the template, and then passing different values for the parameters for each run. See Parameterization of Workflow Templates for more information.

Inline

Workflows can be instantiated inline using the gcloud command with workflow template YAML files or by calling the Cloud Dataproc InstantiateInline API. Inline workflows do not create or modify workflow template resources.

Workflow Templates Use Cases

  • Automation of repetitive tasks. Workflows encapsulate frequently used cluster configurations and jobs.

  • Transactional, Fire and Forget API interaction model. Workflow Templates replace the steps involved in a typical flow, which include creating the cluster, submitting jobs, polling, and deleting the cluster. They use a single single token to track progress from cluster creation to deletion, and automate error handling and recovery. They also simplify the integration of Cloud Dataproc with other tools, such as Cloud Functions and Cloud Composer.

  • Support for ephemeral and long-lived clusters. A common complexity associated with running Apache Hadoop is tuning and right-sizing clusters. Ephemeral (managed) clusters are easier to configure since they run a single workload. Cluster-selectors can be used with longer-lived clusters to repeatedly execute the same workload without incurring the amortized cost of creating and deleting clusters.

  • Granular IAM security. Creating Cloud Dataproc clusters and submitting jobs require all-or-nothing IAM permissions. Workflow Templates use a per-template workflowTemplates.instantiate permission, and do not depend on cluster or job permissions.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataproc Documentation