Run a workflow that executes other workflows in parallel

---

This tutorial shows you how to create and run a parent workflow that executes multiple child workflows in parallel.

In the following diagram, four parallel executions of the child workflow are invoked. This allows the parent workflow to process data in parallel branches, and reduces the overall execution time. The parent workflow waits for all the child workflow executions to finish before returning a summary of successful and failed executions, simplifying any error detection.

Objectives

In this tutorial you will:

1. Create and deploy a child workflow that receives data from a parent workflow.
2. Create and deploy a parent workflow that executes multiple child workflows using a parallel for loop.
3. Run the parent workflow which invokes parallel executions of the child workflow.
4. The results of all successful and failed child workflow executions are stored and returned in a map.

You can run the following commands in the Google Cloud console or by using the Google Cloud CLI in either your terminal or Cloud Shell.

Costs

In this document, you use the following billable components of Google Cloud:

• Workflows

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

Console

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Workflow Executions and Workflows APIs.

   Enable the APIs

5. Create a service account:

   1. In the Google Cloud console, go to the Create service account page.

      Go to Create service account
   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for quickstart.

   4. Click Create and continue.

   5. Grant the Workflows > Workflows Invoker role to the service account.

      To grant the role, find the Select a role list, then select Workflows > Workflows Invoker.

      Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.
   6. Click Continue.

   7. Click Done to finish creating the service account.

6. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

7. Make sure that billing is enabled for your Google Cloud project.

8. Enable the Workflow Executions and Workflows APIs.

   Enable the APIs

9. Create a service account:

   1. In the Google Cloud console, go to the Create service account page.

      Go to Create service account
   2. Select your project.

   3. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

      In the Service account description field, enter a description. For example, Service account for quickstart.

   4. Click Create and continue.

   5. Grant the Workflows > Workflows Invoker role to the service account.

      To grant the role, find the Select a role list, then select Workflows > Workflows Invoker.

      Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.
   6. Click Continue.

   7. Click Done to finish creating the service account.

gcloud

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Workflow Executions and Workflows APIs:

   gcloud services enable workflowexecutions.googleapis.com workflows.googleapis.com

7. Set up authentication:

   1. Create the service account:

      gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

      Replace SERVICE_ACCOUNT_NAME with a name for the service account.

   2. Grant the roles/workflows.invoker IAM role to the service account:

      gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=roles/workflows.invoker

      Replace the following:

      • SERVICE_ACCOUNT_NAME: the name of the service account
      • PROJECT_ID: the project ID where you created the service account
      Note: The --role flag affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.
8. Install the Google Cloud CLI.

9. To initialize the gcloud CLI, run the following command:

   gcloud init

10. Create or select a Google Cloud project.

    Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

11. Make sure that billing is enabled for your Google Cloud project.

12. Enable the Workflow Executions and Workflows APIs:

    gcloud services enable workflowexecutions.googleapis.com workflows.googleapis.com

13. Set up authentication:

    1. Create the service account:

       gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

       Replace SERVICE_ACCOUNT_NAME with a name for the service account.

    2. Grant the roles/workflows.invoker IAM role to the service account:

       gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=roles/workflows.invoker

       Replace the following:

       • SERVICE_ACCOUNT_NAME: the name of the service account
       • PROJECT_ID: the project ID where you created the service account
       Note: The --role flag affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later. In production environments, do not grant the Owner, Editor, or Viewer roles. Instead, grant a predefined role or custom role that meets your needs.

Create and deploy a child workflow

A child workflow can receive and process data from a parent workflow. The child workflow demonstrates this by doing the following:

• Receives an integer as an argument
• Sleeps for 10 seconds to simulate some processing

• Returns an indicator (based on whether the integer is an odd or even number) to simulate success or failure of the workflow execution

Console

1. In the Google Cloud console, go to the Workflows page.

   Go to Workflows

2. Click Create.

3. Enter the name, workflow-child, for the new workflow.

4. In the Region list, select us-central1.

5. Select the Service account you previously created.

6. Click Next.

7. In the workflow editor, enter the following definition for your workflow:

   main:
     params: [args]
     steps:
       - init:
           assign:
             - iteration : ${args.iteration}
       - wait:
           call: sys.sleep
           args:
               seconds: 10
       - check_iteration_even_or_odd:
           switch:
             - condition: ${iteration % 2 == 0}
               next: raise_error
       - return_message:
           return: ${"Hello world"+iteration}
       - raise_error:
           raise: ${"Error with iteration "+iteration}

8. Click Deploy.

gcloud

1. Create a source code file for your workflow:

   touch workflow-child.yaml
   

2. Open your source code file in a text editor and copy the following workflow to the file.

   main:
     params: [args]
     steps:
       - init:
           assign:
             - iteration : ${args.iteration}
       - wait:
           call: sys.sleep
           args:
               seconds: 10
       - check_iteration_even_or_odd:
           switch:
             - condition: ${iteration % 2 == 0}
               next: raise_error
       - return_message:
           return: ${"Hello world"+iteration}
       - raise_error:
           raise: ${"Error with iteration "+iteration}

3. Deploy the workflow:

   gcloud workflows deploy workflow-child \
       --source=workflow-child.yaml \
       --location=us-central1 \
       --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com
   

   Replace SERVICE_ACCOUNT_NAME with the name of the service account you previously created.

Create and deploy the parent workflow

The parent workflow executes multiple branches of the child workflow using a parallel for loop.

1. Copy the source code for the workflow definition. It consists of the following parts:

   1. A map is used to store the results of the child workflow executions. For more information, see Maps.

      main:
        steps:
          - init:
              assign:
                - execution_results: {} # results from each execution
                - execution_results.success: {} # successful executions saved under 'success' key
                - execution_results.failure: {} # failed executions saved under 'failure' key

   2. A for loop is executed in parallel to invoke the child workflow. For more information, see Parallel steps and Iteration.

      - execute_child_workflows:
          parallel:
            shared: [execution_results]
            for:
              value: iteration
              in: [1, 2, 3, 4]
              steps:
                  - iterate:

   3. The child workflow is invoked using a connector. Each iteration of the child workflow is passed the iteration argument. The parent workflow waits for and stores the result of each child workflow execution. For more information, see Workflows Executions API connector and Runtime arguments.

      try:
        steps:
          - execute_child_workflow:
              call: googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run
              args:
                workflow_id: workflow-child
                #location: ...
                #project_id: ...
                argument:
                  iteration: ${iteration}
              result: execution_result
          - save_successful_execution:
              assign:
                - execution_results.success[string(iteration)]: ${execution_result}
      except:
          as: e
          steps:
            - save_failed_execution:
                assign:
                  - execution_results.failure[string(iteration)]: ${e}

   4. The execution results are returned. For more information, see Complete the execution of a workflow.

      - return_execution_results:
          return: ${execution_results}

2. Deploy the workflow:

   Console

   1. In the Google Cloud console, go to the Workflows page:

      Go to Workflows

   2. Click Create.

   3. Enter the name, workflow-parent, for the new workflow.

   4. In the Region list, select us-central1.

   5. Select the Service account you previously created.

   6. Click Next.

   7. In the workflow editor, paste the definition for the parent workflow.

   8. Click Deploy.

   gcloud

   1. Create a source code file for your workflow:

      touch workflow-parent.yaml
      

   2. Open your source code file in a text editor and paste the definition for the parent workflow.

   3. Deploy the workflow:

      gcloud workflows deploy workflow-parent \
          --source=workflow-parent.yaml \
          --location=us-central1 \
          --service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com
      

      Replace SERVICE_ACCOUNT_NAME with the name of the service account you previously created.

Execute the parent workflow

Execute the parent workflow so that invocations of the child workflow run in parallel. The executions should take about about 10 seconds to complete.

Console

1. In the Google Cloud console, go to the Workflows page:

   Go to Workflows

2. On the Workflows page, click the workflow-parent workflow to go to its details page.

3. On the Workflow details page, click Execute.

4. Click Execute again.

5. View the results of the workflow in the Output pane.

   The results should be similar to the following, indicating errors with iterations 2 and 4, and success with iterations 1 and 3.

   "failure": {
     "2": {
       "message": "Execution failed or cancelled.",
       "operation": {
         "argument": "{\"iteration\":2}",
         "duration": "10.157992541s",
         "endTime": "2023-07-11T13:13:13.028424329Z",
         "error": {
           "context": "RuntimeError: \"Error with iteration 2\"\nin step \"raise_error\", routine \"main\", line: 18",
           "payload": "\"Error with iteration 2\"",
   ...
     "4": {
       "message": "Execution failed or cancelled.",
       "operation": {
         "argument": "{\"iteration\":4}",
         "duration": "10.157929734s",
         "endTime": "2023-07-11T13:13:13.061289142Z",
         "error": {
           "context": "RuntimeError: \"Error with iteration 4\"\nin step \"raise_error\", routine \"main\", line: 18",
           "payload": "\"Error with iteration 4\"",
   ...
   "success": {
     "1": "Hello world1",
     "3": "Hello world3"

gcloud

Execute the workflow:

gcloud workflows run workflow-parent \
    --location=us-central1

The results should be similar to the following, indicating errors with iterations 2 and 4, and success with iterations 1 and 3.

Waiting for execution [06c753e4-6947-4c62-ac0b-2a9d53fb1b8f] to complete...done.
argument: 'null'
duration: 14.065415004s
endTime: '2023-07-11T12:50:43.929023883Z'
name: projects/386837416586/locations/us-central1/workflows/workflow-parent/executions/06c753e4-6947-4c62-ac0b-2a9d53fb1b8f
result: '{"failure":{"2":{"message":"Execution failed or cancelled.","operation":{"argument":"{\"iteration\":2}","duration":"10.143718070s","endTime":"2023-07-11T12:50:40.673209821Z","error":{"context":"RuntimeError:
...
"Error with iteration 2\"\nin step \"raise_error\", routine \"main\", line: 18","payload":"\"Error
...
"Error with iteration 4\"\nin step \"raise_error\", routine \"main\", line: 18","payload":"\"Error
...
"success":{"1":"Hello world1","3":"Hello world3"}}'
startTime: '2023-07-11T12:50:29.863608879Z'
state: SUCCEEDED

You have successfully created and deployed a workflow that invokes a child workflow, executes four iterations of the child workflow in parallel branches, and returns an indicator of success or failure for each child workflow execution.

Clean up

If you created a new project for this tutorial, delete the project. If you used an existing project and wish to keep it without the changes added in this tutorial, delete resources created for the tutorial.

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete tutorial resources

Delete the workflows created in this tutorial:

gcloud workflows delete workflow-child
gcloud workflows delete workflow-parent

What's next

• To learn more about the Workflows syntax, see the Workflows syntax reference.
• To learn more about Workflows connectors, see Understand connectors.