To create a workflow, you define the desired steps and order of execution using the Workflows syntax in either YAML or JSON. This page is a reference that specifies the correct syntax for defining workflows.
Key
[]
: Optional{}
: Required|
: Used to separate multiple options...
: Indicates an omission to improve clarity and shorten the length of the example, or that you can add more objects, fields, or arguments
File structure
Workflow files have the following characteristics:
- They contain only one workflow.
- They don't require headers.
- They are either a valid YAML or JSON file when using Workflows with the Cloud SDK.
Steps
Every workflow must have at least one step. By default, Workflows treats steps as if they are in an ordered list and executes them one at a time until all the steps have run. For example, this workflow has two steps:
- STEP_NAME_A: ... - STEP_NAME_B: ...
The step name can include any alphanumeric character as well as underscores.
Workflows currently supports the following types of steps:
- Invoking an HTTP endpoint
- Assigning a variable
- Sleeping
- Creating the logic for conditional jumps
- Returning a value
Invoking an HTTP endpoint
This type of step allows you to make an HTTP request. Both HTTP and HTTPS
requests are supported. The most common HTTP request methods have a built-in
call shortcut (http.get
and http.post
), but you can make any type of HTTP
request by setting the call
field to http.request
and specifying the type of
request using the method
field.
- STEP_NAME: call: {http.get|http.post|http.request} args: url: URL_VALUE [method: REQUEST_METHOD] [headers: KEY:VALUE ...] [body: KEY:VALUE ...] [query: KEY:VALUE ...] [auth: type:{OIDC|OAuth2}] [timeout: VALUE_IN_SECONDS] [result: RESPONSE_VALUE]
call
: Required. Usehttp.get
,http.post
, orhttp.request
for HTTP requests.url
: Required. URL where the request is sent.method
: Required if using call typehttp.request
. The type of HTTP request method to use. For example:GET
POST
PATCH
DELETE
headers
,body
,query
: Optional. Fields to supply input to the API.auth
: Optional. Required if the API being called requires authentication. See Making authenticated requests for more information.timeout
: Optional. Time in seconds. How long a request is allowed to run before throwing an exception.result
: Optional. Variable name where the result of an HTTP invocation step is stored.
Example:
YAML
JSON
Assigning variables
To initialize variables or update existing variables, use a variable assignment step. You can perform up to ten assignments in a single step. Variables can be assigned to a particular value or to the result of an expression.
- STEP_NAME: assign: - VARIABLE_NAME: VALUE
Example:
- assign_vars:
assign:
- number: 5
- number_plus_one: ${number+1}
- other_number: 10
- string: "hello"
Sleeping
To pause the execution of a workflow, use a sleep step:
- STEP_NAME: call: sys.sleep args: seconds: SLEEP_IN_SECONDS
Calls
Many types of workflow steps make calls. For example:
- STEP_NAME: call: ...
Some calls can also be made within an expression. For example:
- initVariables: assign: - project: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
Calls performed during a workflow can be either blocking or non-blocking. Blocking calls are calls that block a workflow's execution; they must be resolved before a workflow execution can complete. Blocking calls cannot be made within an expression.
Blocking calls
The following list contains all the blocking calls in Workflows:
http.*
(All HTTP calls)sys.sleep
sys.log
googleapis.*
(All connector calls)
Jumps
You can use jumps to control what step Workflows will execute next.
Basic jumps
At the end of any step, you can use next
to define what step
Workflows should execute next:
- STEP_NAME: ... next: STEP_NAME_TO_JUMP_TO
Conditional jumps
You can also control the order of a workflow's execution by using a switch
block to jump between steps based on a conditional expression:
- STEP_NAME_A: switch: - condition: ${EXPRESSION_A} next: STEP_NAME_B - condition: ${EXPRESSION_B} next: STEP_NAME_C next: STEP_NAME_D
Each switch
block can include a maximum of 10 conditions. Each expression must
evaluate to true or false.
You can also nest multiple steps inside a switch
block:
- STEP_NAME_A: switch: - condition: ${EXPRESSION} steps: - STEP_NAME_B: ... - STEP_NAME_C: ... next: STEP_NAME_D
Example:
YAML
JSON
See Controlling the order of execution in a workflow for more information about using jumps to define order of execution.
Stopping a workflow's execution
A workflow's execution finishes when an exception is thrown, after all steps
have been executed, or when the keywords end
or return
are used.
Using end
Use next: end
to stop a workflow's execution if you don't need to return
a value:
- STEP_NAME: ... next: end
Use return
to stop a workflow's execution and return a value, variable, or
expression:
- STEP_NAME: ... return: ${VARIABLE}
Variables
You can create and assign variables in a workflow by using an assign step or by assigning them as the result of an HTTP invocation step. You can access HTTP response information stored in a variable using the built-in parser.
Storing a result in a variable
To store the result of a step to variable:
- STEP_NAME: ... result: VARIABLE
Accessing HTTP response data saved in a variable
When a response of type application/json
is stored in a variable, the JSON
response is converted to a dictionary you access as a response body.
Workflows includes a built-in parser for accessing this data. To access
the fields from the HTTP response, use the following syntax:
${VARIABLE_NAME.body|code.PATH_TO_FIELD}
- VARIABLE_NAME: The name of the workflow variable where you saved a JSON response.
body
: Use thebody
field to access the body of the HTTP response. Use thecode
field to access the HTTP response code.- PATH_TO_FIELD: The path to the field in the JSON response that you
want to access. May be simply the name of the field, or if the field is nested
inside an object, it may take the form of
object1.object2.field
.
For example, if an API returns {"age":50}
and a workflow stores that response
in a variable called age_response
, the following:
age_response.body.age
returns the value of the age
field; in this case, 50
.
Arrays
Workflows supports arrays for storing data. Arrays can be defined in a workflow in two different ways. This section also covers the syntax for iterating through an array.
Array definition
Arrays can be created in a workflow by being returned as the result of a step
or by being defined in an assign
step.
Array returned by a step
HTTP requests often return a JSON response as an array that we store as a
variable in the workflow. In the following example, we make a request to an
example API that returns a list of months. We store the array as a variable named
monthsList
and then return the third element of the array:
- step_a:
call: http.get
args:
url: https://somewhere.com/getMonths
result: monthsList
- step_b:
return: ${monthsList.body[2]}
Array defined in an assign step
To define an array, use an assign
step:
- step_a:
assign:
- num_array: ["zero","one","two"]
Iterating through an array
You can use a combination of conditional jumps, variables, and the len()
function to iterate through the array. For example:
YAML
JSON
Expressions
Expressions are evaluated by the workflow engine and the output is used at the time of execution, such as assigning the result of an expression to a variable or returning the result of an expression.
All expressions must begin with a $
and be enclosed in curly brackets:
${EXPRESSION}
You can use expressions for the following:
- Conditional jumps
- Assigning values to variables
- HTTP request details, including all parameters in the
body
,header
,query
, andurl
sections - Retry values, including
max_retries
andmultiplier
- Returning values from a workflow step
The Workflows syntax supports the following elements in the definition of an expression:
[0...9]
: numbers""
: strings-
(minus sign): indicates negative numbers.
(dot): indicates decimal place+
: arithmetic addition and string concatenation-
: arithmetic subtraction and negation*
: arithmetic multiplication/
: float division%
: remainder division//
: floor division()
: parenthesesand
(case sensitive): logical ANDor
(case sensitive): logical ORnot
(case sensitive): logical NOTvariableName
: reference a variableobject.field
: reference a value in an objectarray[index]
: reference an index in an array
Runtime arguments
You can access data passed at runtime by adding a params
field to your
workflow, which names the dictionary the workflow uses to store the data you
pass in. You can then use dot notation to access the arguments:
main:
params: [DICTIONARY_NAME]
steps:
- step1:
return: ${DICTIONARY_NAME.PARAM_NAME}
For more information about using runtime arguments, see Passing runtime arguments in an execution request.
Data types
The Workflows syntax supports the following data types:
- Integer (64 bit, signed)
- Double (64 bit, signed floating point number)
- String (supports unicode, <= 64kB length)
- Boolean (true/false, True/False, TRUE/FALSE)
- Null
Conversion functions
The following conversion functions are supported:
double()
: Accepts an attribute of type string or integer and returns a double.int()
: Accepts an attribute of type string or double and returns an integer.string()
: Accepts an attribute of type integer, double, or boolean and returns a string.
Logical operators
The Workflows syntax supports and
, or
, and not
as logical operators.
Logical operators can only be used on boolean values, or on expressions that
evaluate to boolean values.
Implicit data type conversions
These tables show the outcome of expressions using the given operator and the
shown data types as inputs. For example, using the +
operator on two strings
results in a string.
+ operator |
||||
---|---|---|---|---|
INTEGER |
DOUBLE |
STRING |
BOOL |
|
INTEGER |
INTEGER | DOUBLE | ERROR | ERROR |
DOUBLE |
DOUBLE | DOUBLE | ERROR | ERROR |
STRING |
ERROR | ERROR | STRING | ERROR |
BOOL |
ERROR | ERROR | ERROR | ERROR |
In the case of int or double addition, overflow can occur if the resulting value exceeds the available precision. During string concatenation, if the resulting string exceeds the allowed length of the string data type, the operation throws an error.
- operator |
||||
---|---|---|---|---|
INTEGER | DOUBLE | STRING | BOOL | |
INTEGER | INTEGER | DOUBLE | ERROR | ERROR |
DOUBLE | DOUBLE | DOUBLE | ERROR | ERROR |
STRING | ERROR | ERROR | ERROR | ERROR |
BOOL | ERROR | ERROR | ERROR | ERROR |
* operator |
||||
---|---|---|---|---|
INTEGER | DOUBLE | STRING | BOOL | |
INTEGER | INTEGER | DOUBLE | ERROR | ERROR |
DOUBLE | DOUBLE | DOUBLE | ERROR | ERROR |
STRING | ERROR | ERROR | ERROR | ERROR |
BOOL | ERROR | ERROR | ERROR | ERROR |
/ operator |
||||
---|---|---|---|---|
INTEGER | DOUBLE | STRING | BOOL | |
INTEGER | DOUBLE | DOUBLE | ERROR | ERROR |
DOUBLE | DOUBLE | DOUBLE | ERROR | ERROR |
STRING | ERROR | ERROR | ERROR | ERROR |
BOOL | ERROR | ERROR | ERROR | ERROR |
< , > , <= , >= comparison operators |
||||
---|---|---|---|---|
INTEGER | DOUBLE | STRING | BOOL | |
INTEGER | BOOL | BOOL | ERROR | ERROR |
DOUBLE | BOOL | BOOL | ERROR | ERROR |
STRING | ERROR | ERROR | ERROR | ERROR |
BOOL | ERROR | ERROR | ERROR | ERROR |
== , != comparison operators |
|||||
---|---|---|---|---|---|
INTEGER | DOUBLE | STRING | BOOL | NULL | |
INTEGER | BOOL | BOOL | ERROR | ERROR | BOOL |
DOUBLE | BOOL | BOOL | ERROR | ERROR | BOOL |
STRING | ERROR | ERROR | BOOL | ERROR | BOOL |
BOOL | ERROR | ERROR | ERROR | BOOL | BOOL |
NULL | BOOL | BOOL | BOOL | BOOL | BOOL |
Dictionaries
Workflows supports dictionaries that can hold a user-defined structure of variables or arrays.
Dictionary definition
To define a dictionary, add an assign step to the workflow:
- STEP_NAME_A: assign: - DICTIONARY_A: KEY_1: VALUE_1 KEY_2: KEY_3: VALUE_2
For example:
- createDictionary: assign: - myDictionary: FirstName: "John" LastName: "Smith" Age: 26 Address: Street: "Flower Road 12" City: "Superville" Country: "Atlantis"
Reading dictionary values
To read the values in a dictionary, use the following structure within an expression:
${DICTIONARY.KEY}
Use an additional period to access the value of a nested key.
For example, to return the value of Country
from the previous
example:
- lastStep: return: ${myDictionary.Address.Country}
Checking existence of a key in a dictionary
To check whether a given key is present in a dictionary, use the following expression:
${KEY in DICTIONARY}
For example:
- MyStep: switch: - condition: ${"Age" in myDictionary} next: AgeExists
To check whether a key is not in a dictionary, use the not()
function:
- MyStep: switch: - condition: ${not("Age" in myDictionary)} next: AgeMissing
Subworkflows
You can use subworkflows to define a piece of logic that can be called from the main workflow, similar to a routine or function in a programming language. They allow you to repeat steps in a workflow without duplicating the steps and increasing the number of lines in the workflow's definition. Subworkflows can accept input parameters and return values.
If a workflow has a subworkflow, the main workflow must be placed in a main
block. Subworkflows are always defined after the main body of the workflow
definition:
main: steps: - STEP_NAME: ... ... SUBWORKFLOW_NAME: [params: [PARAMETER_1,PARAMETER_2...]] steps: - SUBWORKFLOW_STEP_NAME: ...
You call a subworkflow using the call
field within a workflow step:
call: SUBWORKFLOW_NAME
This example defines a subworkflow and calls it from the main workflow:
main:
steps:
- call_subworkflow:
call: name_message
args:
first_name: "Sherlock"
last_name: "Holmes"
result: output
- return_message:
return: ${output}
name_message:
params: [first_name, last_name]
steps:
- prepMessage:
return: ${"Hello " + first_name + " " + last_name}
Default paremeters in subworkflows
Subworkflows support default values for parameters. Default parameter value is used as only if the parameter wasn't provided as part of a subworkflow call.
SUBWORKFLOW_NAME: [params: [PARAMETER_1, PARAMETER_2: DEFAULT_VALUE2...]] steps: - SUBWORKFLOW_STEP_NAME: ...
For example, the following workflow can be called with or without Country parameter, and if the Country is not specified, it deefaults to "United States".
build_address: params: [Street, ZipCode, Country: "United States"] steps: - concatenate: return: ${Street + ", " + ZipCodee + ", " + Country}
For more information about working with subworkflows, see Creating and using subworkflows.
Error handling
Workflows can retry a step when it encounters an exception instead of failing and ending the execution attempt. The Workflows syntax has several types of built-in error handling strategies:
- Raising exceptions
- Catching and handling HTTP request errors
- Retrying a failed step
Raising exceptions
Raising an exception stops the execution with a fail state and returns the exception as an error message. Exceptions can be either a string or a dictionary.
String
- step_a:
raise: "Something went wrong."
Dictionary
- step_a:
raise:
code: 55
message: "Something went wrong."
Catching and handling HTTP request errors
Workflows considers any HTTP request that returns status code 400
or above
failed. This makes the workflow execution fail unless the workflow catches
and handles the error. Workflows uses a try/except structure for
error handling:
- STEP_NAME: try: call: http.get ... except: as: error_dictionary steps: ...
error_dictionary
: Name of a dictionary variable that contains the
error message. For HTTP requests, the error dictionary has the following
attributes:
code
: HTTP status codemessage
: Human-readable error messagetags
: Error tags. Tags can be any of the following strings:HttpError
: HTTP response received with status code >=400
ConnectionError
: Error connecting to the API endpoint. For example, due to incorrect domain name, DNS resolution problem, etc.TimeoutError
: Timeout value reached without receiving a response
For example:
- read_item:
try:
call: http.get
args:
url: https://host.com/api
result: api_response
except:
as: e
steps:
- known_errors:
switch:
- condition: ${not("HttpError" in e.tags)}
return: "Connection problem."
- condition: ${e.code == 404}
return: "Sorry, URL wasn’t found."
- condition: ${e.code == 403}
return: "Authentication error."
- unhandled_exception:
raise: ${e}
- url_found:
return: ${api_response.body}
The try
block can contain multiple steps, allowing them to share the same
except
block for error handling:
- read_item:
try:
steps:
- step_a:
call: http.get
args:
url: https://host.com/api
result: api_response1
- step_b:
call: http.get
args:
url: https://host.com/api2
result: api_response2
...
except:
as: e
steps:
- KnownErrors:
...
Retrying steps
You can retry steps that return a specific error code, for example, particular HTTP status codes. The retry syntax allows you to:
- Define the maximum number of retry attempts
- Define a backoff model to increase the likelihood of success
Workflows has default retry policies available for both idempotent and non-idempotent steps. Additionally, the default retry policies can be modified, and you can create a custom retry policy if the existing ones don't work for your use case. If you use a default retry policy, you don't need to specify a predicate or define the retry configuration values.
A retry policy is composed of a predicate that defines which error codes should be retried and default values for the retry configuration values.
- step_name:
try:
steps:
...
retry: [${http.default_retry} | ${http.default_retry_non_idempotent}]
[predicate: ${http.default_retry_predicate} | ${http.default_retry_predicate_non_idempotent}]
[max_retries: number_of_retries]
[backoff:
initial_delay: delay_seconds
max_delay: max_delay_seconds
multiplier: delay_multiplier]
- retry: Optional. If omitted, all other fields are required. Options
include
${http.default_retry}
and${http.default_retry_non_idempotent}
. Allows you to specify a default retry policy to use. If you specify a retry policy, omit all other fields in the retry block. - predicate: Required if you don't select a default retry policy. Defines
which error codes will be retried. Options include
${http.default_retry_predicate}
,${http.default_retry_predicate_non_idempotent}
, or a custom predicate defined as a subworkflow. Retry configuration values: Required if a default retry policy is not used.
- max_retries: Maximum number of times a step will be retried.
backoff: Block that controls how retries occur. Has the following parameters:
- initial_delay: Delay in seconds between the initial failure and the first retry.
- max_delay: Maximum delay in seconds between retries.
- delay_multiplier: Multiplier applied to the previous delay to calculate the delay for the subsequent retry.
For example, given the following retry configuration values:
max_retries: 8 backoff: initial_delay: 1 max_delay: 60 multiplier: 2
The step will be retried a total of eight times. The initial delay is 1 second, and the delay is doubled on each attempt, with a maximum delay of 60 seconds. In this case, the delays between subsequent attempts are: 1, 2, 4, 8, 16, 32, 60, and 60 (time given in seconds). After eight attempts, the step is considered failed and an exception is raised.
You can configure the retry
block in one of three ways:
Using a default retry policy
There are two default retry policies available: one for idempotent steps, and one for non-idempotent steps.
The default retry policies are composed of a default retry predicate and a set of default retry configuration values.
Default retry policy for idempotent steps
Note: You should only use this retry policy for idempotent steps (steps that can be safely repeated.)
The default retry policy for idempotent steps has the following configuration:
predicate
:${http.default_retry_predicate}
. Retries HTTP status codes[429, 502, 503, 504]
, connection error, or timeoutmax_retries
: 5initial_delay
: 1.0max_delay
: 60multiplier
: 1.25
For example, to use the default retry policy for an idempotent step:
- idempotent_step:
try:
call: http.get
args:
url: https://host.com/api
retry: ${http.default_retry}
Default retry for non-idempotent steps
Note: You should only use this retry policy for non-idempotent steps (steps that can't be safely repeated.)
The default retry policy for non-idempotent steps has the following configuration:
predicate
:${http.default_retry_predicate_non_idempotent}
. Retries HTTP status codes[429, 503]
max_retries
: 5initial_delay
: 1.0max_delay
: 60multiplier
: 1.25
For example, to use the default retry policy for a non-idempotent step:
- non_idempotent_step:
try:
call: http.get
args:
url: https://host.com/api
retry: ${http.default_retry_non_idempotent}
Using a default retry predicate with custom retry configuration
Select the appropriate predicate for the type of step (idempotent or non-idempotent), then supply the desired retry configuration values. For example, the following retry policy uses the default predicate for non-idempotent steps and defines custom configuration values:
- step_name:
try:
steps:
...
retry:
predicate: ${http.default_retry_predicate_non_idempotent}
max_retries: 10
backoff:
initial_delay: 1
max_delay: 90
multiplier: 3
Creating a custom retry policy
To create a custom retry policy, use a subworkflow to define your predicate
(the set of errors the policy will be called for). Then call your predicate and
define your desired retry configuration values in the retry
block in the main
workflow.
main:
- step_name:
try:
steps:
...
retry:
predicate: ${retry_predicate}
max_retries: number_of_retries
backoff:
initial_delay: delay_seconds
max_delay: max_delay_seconds
multiplier: delay_multiplier
retry_predicate:
params: [e]
steps:
...
For example, the following code implements a custom retry policy that only retries HTTP requests that returned HTTP status code 500:
YAML
JSON
What's next
Get started with Workflows by following a quickstart.
Learn how to create a workflow and update an existing workflow.
Learn how to execute your workflow.