Developers & Practitioners

Deploying multi-YAML Workflows definitions with Terraform

May 12, 2021

https://storage.googleapis.com/gweb-cloudblog-publish/images/appdev.max-2600x2600.jpg

Mete Atamel

Cloud Developer Advocate

I’m a big fan of using Workflows to orchestrate and automate services running on Google Cloud and beyond. In Workflows, you can define a workflow in a YAML or JSON file and deploy it using gcloud or using Google Cloud Console. These approaches work but a more declarative and arguably better approach is to use Terraform.

Let’s see how to use Terraform to define and deploy workflows and explore options for keeping Terraform configuration files more manageable.

Single Terraform file

Terraform has a google_workflows_workflow resource to define and deploy workflows. For step-by-step instructions, see our basic Terraform sample, which shows how to define a workflow in main.tf and how to deploy it using Terraform.

Let’s take a closer look at how the Workflows resource is defined in Terraform:

resource "google_workflows_workflow" "workflows_example" {
  name            = "sample-workflow"
  region          = var.region
  description     = "A sample workflow"
  service_account = google_service_account.workflows_service_account.id
  source_contents = <<-EOF
  # This is a sample workflow, feel free to replace it with your source code
  #
  # This workflow does the following:
  # - reads current time and date information from an external API and stores
  #   the response in CurrentDateTime variable
  # - retrieves a list of Wikipedia articles related to the day of the week
  #   from CurrentDateTime
  # - returns the list of articles as an output of the workflow
  # FYI, In terraform you need to escape the $$ or it will cause errors.
  - getCurrentTime:
      call: http.get
      args:
          url: https://us-central1-workflowsample.cloudfunctions.net/datetime
      result: CurrentDateTime
  - readWikipedia:
      call: http.get
      args:
          url: https://en.wikipedia.org/w/api.php
          query:
              action: opensearch
              search: $${CurrentDateTime.body.dayOfTheWeek}
      result: WikiResult
  - returnOutput:
      return: $${WikiResult.body[1]}
EOF

}

You can see that everything about the workflow, such as name, region, service account, and even the workflow definition itself are defined in this single file. While this is workable for simple workflow definitions, it’s hardly maintainable for larger workflows.

Importing a Workflow definition file

A better approach is to keep the workflow definition in a separate YAML file and import that into Terraform. The templatefile function of Terraform makes this possible — and in fact very easy to do.

In the Terraform with imported YAML sample, you can see how to import an external workflows.yaml file into your Terraform definition:

Importing multiple Workflow definition files

Importing the workflow YAML file is a step in the right direction, but in large workflow definitions, you often have a main workflow calling multiple subworkflows. Workflows doesn’t currently support importing or merging workflow and subworkflow definitions. You end up with a single workflow definition file for the main workflow and all the subworkflows. This is not maintainable.

Ideally, you’d have each subworkflow in its own file and have the main workflow simply refer to them. Thankfully, this is easy to do in Terraform.

In the Workflows Terraform with multiple external YAMLs sample, you can see how to import an external workflows.yaml file for the main workflow and a subworkflow.yaml file for the subworkflow into your Terraform definition:

This is more maintainable for sure! One minor issue is that all YAMLs do end up getting merged and deployed as a single YAM to Workflows. When you debug your workflows and subworkflows, you might get confused with line numbers of your subworkflows.

This wraps up our discussion of Workflows and Terraform. You can check out our workflows-demos repo for all the source code for Terraform samples and more. Thanks to Jamie Thomson for the template file idea on Terraform. Please reach out to me on Twitter @meteatamel for any questions or feedback.

Developers & Practitioners