The Dataproc WorkflowTemplates API provides a flexible and easy-to-use mechanism for managing and executing workflows. A Workflow Template is a reusable workflow configuration. It defines a graph of jobs with information on where to run those jobs.
- Instantiating a Workflow Template launches a Workflow. A Workflow is an operation that runs a Directed Acyclic Graph (DAG) of jobs on a cluster. If the workflow uses a managed cluster, it creates the cluster, runs the jobs, and then deletes the cluster when the jobs are finished.
- If the workflow uses a cluster selector, it runs jobs on a selected existing cluster.
- Workflows are ideal for complex job flows. You can create job dependencies so that a job starts only after its dependencies complete successfully.
- Creating a workflow template does not create a Dataproc cluster or submit jobs. Clusters and jobs associated with workflows are created when a workflow template is instantiated.
Kinds of workflow templates
A workflow template can specify a managed cluster. The workflow will create this "ephemeral" cluster to run workflow jobs, and then delete the cluster when the workflow is finished.
A workflow template can specify an existing cluster on which to run workflow jobs by specifying one or more user labels that were previously applied to one or more clusters. The workflow will run on a cluster that matches all of the specified labels. If multiple clusters match the label(s), Dataproc will select the cluster with the most YARN available memory to run all workflow jobs. At the end of workflow, the selected cluster is not deleted. See Using cluster selectors with workflows for more information.
If you will run a workflow template multiple times with different values, you can avoid editing the workflow template for each run by defining parameters in the template, and then passing different values for the parameters for each run. See Parameterization of Workflow Templates for more information.
Workflows can be instantiated inline using the
gcloud command with
workflow template YAML files or by calling the Dataproc
API. Inline workflows do not create or modify workflow template resources.
Workflow Templates Use Cases
Automation of repetitive tasks. Workflows encapsulate frequently used cluster configurations and jobs.
Transactional, Fire and Forget API interaction model. Workflow Templates replace the steps involved in a typical flow, which include creating the cluster, submitting jobs, polling, and deleting the cluster. They use a single token to track progress from cluster creation to deletion, and automate error handling and recovery. They also simplify the integration of Dataproc with other tools, such as Cloud Functions and Cloud Composer.
Support for ephemeral and long-lived clusters. A common complexity associated with running Apache Hadoop is tuning and right-sizing clusters. Ephemeral (managed) clusters are easier to configure since they run a single workload. Cluster-selectors can be used with longer-lived clusters to repeatedly execute the same workload without incurring the amortized cost of creating and deleting clusters.
Granular IAM security. Creating Dataproc clusters and submitting jobs require all-or-nothing IAM permissions. Workflow Templates use a per-template workflowTemplates.instantiate permission, and do not depend on cluster or job permissions.