Understand Workflows

Workflows is a fully-managed orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.

By incorporating Workflows into solutions, you can make service dependencies explicit and observable end-to-end. A workflow that specifies an application, operational, or business process provides a source-of-truth or canonical narrative for the process.

Workflows is serverless, scaling up as needed, and no charges are incurred while idle. Because a workflow contains no code or library dependencies, it does not require security patches. Once you deploy a workflow, you can expect it to reliably execute without maintenance. A workflow can hold state, retry, poll, or wait for up to a year.

The following diagram shows an example of using Workflows to orchestrate services:

Service orchestration using Workflows

Key use cases

Workflows supports many use cases. Some examples are:

Service orchestration Create solutions by combining services—Perform a sequence of operations across multiple systems, waiting for all operations to complete. Can be event-driven. For example:
  • Send newly uploaded files to Cloud Vision AI, then write tags into Firestore
  • Call a Cloud Function and send the results to a Cloud Run service
Batch jobs Operate on multiple items—Perform operations on a set of items or batch data. Often scheduled. For example:
  • Send daily customer emails
  • Prepare and run BigQuery or machine learning jobs
  • Generate reports
Business processes Automate line-of-business workflows—Encode the steps in a business process, including conditions, actions, and human-in-the-loop events. For example:
  • Track an order from request to fulfillment
  • Automate resource requests with approvals
Automation Managed execution of service operations—Easily script sequences of Google Cloud service operations. For example:
  • Provision new tenant projects or infrastructure
  • Turn down resources on a schedule or through event triggers

Core concepts

A workflow consists of a series of steps described using the Workflows syntax, and can be written in either YAML or JSON. This is the workflow's definition. For a detailed explanation of the Workflows syntax, see the Syntax reference.

After a workflow is created, it is deployed, which makes the workflow ready for execution. Learn how to create and update a workflow in the Google Cloud Console or by using the gcloud command-line tool.

An execution is a single run of the logic contained in a workflow's definition. A workflow that hasn't been executed generates no charges. All workflow executions are independent, and the product's rapid scaling allows for a high number of concurrent executions. You can execute a workflow using the client libraries, in the Cloud Console, using the gcloud command-line tool, or by sending a request to the Workflows REST API. For details, see Execute a workflow.

Key capabilities

Execution control

Steps
To create a workflow, you define the desired steps and order of execution using the Workflows syntax. Every workflow must have at least one step. By default, Workflows treats steps as if they are in an ordered list and executes them one at a time until all the steps have run. For details, see Steps.
Conditions
You can use a switch block as a selection mechanism that allows the value of an expression to control the flow of a workflow's execution. For details, see Conditions.
Iteration
You can use a for loop to iterate over a sequence of numbers or through a collection of data, such as a list or map. For details, see Iteration.
Subworkflows
A subworkflow works similarly to a routine or function in a programming language, allowing you to encapsulate a step or set of steps that your workflow will repeat multiple times. For details, see Subworkflows.

Triggering executions

Manual
You can manage workflows from either the Google Cloud Console or from the command line using the Cloud SDK. Visualization support while editing the Workflows syntax is also available through the Google Cloud Console.
Programmatic
The Cloud Client Libraries for the Workflows API, or the REST API, can be used to manage workflows. For details, see Workflows APIs and reference.
Scheduled
You can use Cloud Scheduler to run a workflow on a particular schedule, such as every Monday at 9 AM or every 15 minutes. For details, see Schedule a workflow using Cloud Scheduler.
Runtime arguments
Data passed at runtime can be accessed by adding a params field to your main workflow (placed in a main block). The params field names the map the workflow uses to store the data you pass in. You can then use dot notation to access the arguments. For details, see Runtime arguments.

Connecting services

HTTP APIs
You can define a workflow step that makes an HTTP call and assign the response from the call to a variable. For example, you can invoke a Google Cloud service such as Cloud Functions or Cloud Run through an HTTP request. Both HTTP and HTTPS requests are supported. For details, see Make an HTTP request.
Connectors
Workflows publishes connectors that can be used to connect to other Google Cloud APIs within a workflow, and to integrate your workflows with those Google Cloud products. They simplify calling services because they handle the formatting of requests for you, and provide methods and arguments so that you don't need to know the details of a Google Cloud API. For details, see Understand connectors.
Standard library and environment variables
The Workflows standard library and built-in environment variables allow you to easily construct arguments for services and process responses.
The standard library includes modules and frequently used functions, such as for data type and format conversions. There is no need to import or load libraries in a workflow—library functions work out of the box. For details, see the Standard library overview.
You can access a workflow's environment information (such as its location or project identifier) using built-in environment variables. Built-in environment variables require no declaration and are available in every workflow execution. For details, see Built-in environment variables.

Error handling

You can make your workflows resilient and customize their behavior when a failure occurs by using Workflows' exception handling, including automated HTTP call retries with exponential back-offs, custom error handlers, and other advanced features. For details, see Workflow errors.

Waiting

Callbacks allow workflow executions to wait for another service to make a request to the callback endpoint; that request resumes the execution of the workflow. With callbacks, you can signal to your workflow that a specified event has occurred, and wait on that event without polling. For details, see Wait using callbacks.

You can pause the execution of a workflow by adding a sleep step to your workflow's definition. You can then use sys.sleep to poll for data over a given interval. For details, see Wait using polling.

Authentication and access control

Since every workflow execution requires an authenticated call, you can mitigate the risk of accidental or malicious calls by using Workflows. You can also simplify interactions with other Google Cloud APIs by using IAM-based service accounts, and you can securely store keys and passwords for authenticating to external APIs by using Secret Manager. For details, see Authentication and Access control.

Observability

Workflows automatically generates execution logs for workflow executions in Cloud Logging. You can also control when logs are sent to Logging during a workflow execution through call logging or custom logs. For details, see Send logs to Cloud Logging.

Audit logging information is available through Cloud Audit Logs. For details, see Workflows audit logging information.

Code samples

You can find many useful Workflows code samples on the samples page.

What's next