Syntax reference

To create a workflow, you define the desired steps and order of execution using the Workflows YAML-based syntax. This page is a reference that specifies the correct syntax for defining workflows.

Key

  • []: Optional
  • {}: Required
  • |: Used to separate multiple options
  • ...: Indicates an omission to improve clarity and shorten the length of the example, or that you can add more objects, fields, or arguments

File structure

Workflow files have the following characteristics:

  • They contain only one workflow.
  • They don't require headers.
  • They are saved as a YAML file with the .yaml extension when using Workflows with the Cloud SDK.

Steps

Every workflow must have at least one step. By default, Workflows treats steps as if they are in an ordered list and executes them one at a time until all the steps have run. For example, this workflow has two steps:

- STEP_NAME_A:
    ...
- STEP_NAME_B:
    ...

The step name can include any alphanumeric character as well as underscores.

Workflows currently supports the following types of steps:

  • Invoking an HTTP endpoint
  • Assigning a variable
  • Sleeping
  • Creating the logic for conditional jumps
  • Returning a value

Invoking an HTTP endpoint

This type of step allows you to make an HTTP request. Both HTTP and HTTPS requests are supported. The most common HTTP request methods have a built-in call shortcut (http.get and http.post), but you can make any type of HTTP request by setting the call field to http.request and specifying the type of request using the method field.

- STEP_NAME:
    call: {http.get|http.post|http.request}
    args:
        url: URL_VALUE
        [method: REQUEST_METHOD]
        [headers:
            KEY:VALUE
            ...]
        [body:
            KEY:VALUE
            ...]
        [query:
            KEY:VALUE
            ...]
        [auth:
            type:{OIDC|OAuth2}]
        [timeout: VALUE_IN_SECONDS]
    [result: RESPONSE_VALUE]
  • call: Required. Use http.get, http.post, or http.request for HTTP requests.
  • url: Required. URL where the request is sent.
  • method: Required if using call type http.request. The type of HTTP request method to use. For example:
    • GET
    • POST
    • PATCH
    • DELETE
  • headers, body, query: Optional. Fields to supply input to the API.
  • auth: Optional. Required if the API being called requires authentication. See Making authenticated requests for more information.
  • timeout: Optional. Time in seconds. How long a request is allowed to run before throwing an exception.
  • result: Optional. Variable name where the result of an HTTP invocation step is stored.

Example:

- getCurrentTime:
    call: http.get
    args:
      url: https://us-central1-workflowsample.cloudfunctions.net/datetime
    result: currentTime
- readWikipedia:
    call: http.get
    args:
      url: https://en.wikipedia.org/w/api.php
      query:
        action: opensearch
        search: ${currentTime.body.dayOfTheWeek}
    result: wikiResult
- returnResult:
    return: ${wikiResult.body[1]}

Assigning variables

To initialize variables or update existing variables, use a variable assignment step. You can perform up to ten assignments in a single step. Variables can be assigned to a particular value or to the result of an expression.

- STEP_NAME:
    assign:
        - VARIABLE_NAME: VALUE

Example:

- assign_vars:
    assign:
        - number: 5
        - number_plus_one: ${number+1}
        - other_number: 10
        - string: "hello"

Sleeping

To pause the execution of a workflow, use a sleep step:

- STEP_NAME:
    call: sys.sleep
    args:
        seconds: SLEEP_IN_SECONDS

Jumps

You can use jumps to control what step Workflows will execute next.

Basic jumps

At the end of any step, you can use next to define what step Workflows should execute next:

- STEP_NAME:
    ...
    next: STEP_NAME_TO_JUMP_TO

Conditional jumps

You can also control the order of a workflow's execution by using a switch block to jump between steps based on a conditional expression:

- STEP_NAME_A:
    switch:
        - condition: ${EXPRESSION_A}
          next: STEP_NAME_B
        - condition: ${EXPRESSION_B}
          next: STEP_NAME_C
    next: STEP_NAME_D

Each switch block can include a maximum of 10 conditions. Each expression must evaluate to true or false.

You can also nest multiple steps inside a switch block:

- STEP_NAME_A:
    switch:
        - condition: ${EXPRESSION}
          steps:
              - STEP_NAME_B:
                  ...
              - STEP_NAME_C:
                  ...
    next: STEP_NAME_D

Example:

- getCurrentTime:
    call: http.get
    args:
      url: https://us-central1-workflowsample.cloudfunctions.net/datetime
    result: currentTime
- conditionalSwitch:
    switch:
      - condition: ${currentTime.body.dayOfTheWeek == "Friday"}
        next: friday
      - condition: ${currentTime.body.dayOfTheWeek == "Saturday" OR currentTime.body.dayOfTheWeek == "Sunday"}
        next: weekend
    next: workWeek
- friday:
    return: "It's Friday! Almost the weekend!"
- weekend:
    return: "It's the weekend!"
- workWeek:
    return: "It's the work week."

See Controlling the order of execution in a workflow for more information about using jumps to define order of execution.

Stopping a workflow's execution

A workflow's execution finishes when an exception is thrown, after all steps have been executed, or when the keywords end or return are used.

Using end

Use next: end to stop a workflow's execution if you don't need to return a value:

- STEP_NAME:
    ...
    next: end

Using return

Use return to stop a workflow's execution and return a value, variable, or expression:

- STEP_NAME:
    ...
    return: ${VARIABLE}

Variables

You can create and assign variables in a workflow by using an assign step or by assigning them as the result of an HTTP invocation step. You can access HTTP response information stored in a variable using the built-in parser.

Storing a result in a variable

To store the result of a step to variable:

- STEP_NAME:
    ...
    result: VARIABLE

Accessing HTTP response data saved in a variable

When a response of type application/json is stored in a variable, the JSON response is converted to a dictionary you access as a response body. Workflows includes a built-in parser for accessing this data. To access the fields from the HTTP response, use the following syntax:

${VARIABLE_NAME.body|code.PATH_TO_FIELD}
  • VARIABLE_NAME: The name of the workflow variable where you saved a JSON response.
  • body: Use the body field to access the body of the HTTP response. Use the code field to access the HTTP response code.
  • PATH_TO_FIELD: The path to the field in the JSON response that you want to access. May be simply the name of the field, or if the field is nested inside an object, it may take the form of object1.object2.field.

For example, if an API returns {"age":50} and a workflow stores that response in a variable called age_response, the following:

age_response.body.age

returns the value of the age field; in this case, 50.

Arrays

Workflows supports arrays for storing data. Arrays can be defined in a workflow in two different ways. This section also covers the syntax for iterating through an array.

Array definition

Arrays can be created in a workflow by being returned as the result of a step or by being defined in an assign step.

Array returned by a step

HTTP requests often return a JSON response as an array that we store as a variable in the workflow. In the following example, we make a request to an example API that returns a list of months. We store the array as a variable named monthsList and then return the third element of the array:

- step_a:
    call: http.get
    args:
        url: https://somewhere.com/getMonths
    result: monthsList
- step_b:
    return: ${monthsList.body[2]}

Array defined in an assign step

To define an array, use an assign step:

- step_a:
    assign:
        - num_array: ["zero","one","two"]

Iterating through an array

You can use a combination of conditional jumps, variables, and the len() function to iterate through the array. For example:

- define:
    assign:
      - array: ["foo", "ba", "r"]
      - result: ""
      - i: 0
- check_condition:
    switch:
      - condition: ${len(array) > i}
        next: iterate
    next: exit_loop
- iterate:
    assign:
      - result: ${result + array[i]}
      - i: ${i+1}
    next: check_condition
- exit_loop:
    return:
        concat_result: ${result}

Expressions

Expressions are evaluated by the workflow engine and the output is used at the time of execution, such as assigning the result of an expression to a variable or returning the result of an expression.

All expressions must begin with a $ and be enclosed in curly brackets:

   ${EXPRESSION}

You can use expressions for the following:

The Workflows syntax supports the following elements in the definition of an expression:

  • [0...9]: numbers
  • "": strings
  • - (minus sign): indicates negative numbers
  • . (dot): indicates decimal place
  • +: arithmetic addition and string concatenation
  • -: arithmetic subtraction and negation
  • *: arithmetic multiplication
  • /: float division
  • %: remainder division
  • //: floor division
  • (): parentheses
  • and (case sensitive): logical AND
  • or (case sensitive): logical OR
  • not (case sensitive): logical NOT
  • variableName: reference a variable
  • object.field: reference a value in an object
  • array[index]: reference an index in an array

Runtime arguments

You can access data passed at runtime by adding a params field to your workflow, which names the dictionary the workflow uses to store the data you pass in. You can then use dot notation to access the arguments:

main:
    params: [DICTIONARY_NAME]
    steps:
      - step1:
          return: ${DICTIONARY_NAME.PARAM_NAME}

For more information about using runtime arguments, see Passing runtime arguments in an execution request.

Data types

The Workflows syntax supports the following data types:

  • Integer (64 bit, signed)
  • Double (64 bit, signed floating point number)
  • String (supports unicode, <= 64kB length)
  • Boolean (true/false, True/False, TRUE/FALSE)
  • Null

Conversion functions

The following conversion functions are supported:

  • double(): Accepts an attribute of type string or integer and returns a double.

  • int(): Accepts an attribute of type string or double and returns an integer.

  • string(): Accepts an attribute of type integer, double, or boolean and returns a string.

Logical operators

The Workflows syntax supports and, or, and not as logical operators. Logical operators can only be used on boolean values, or on expressions that evaluate to boolean values.

Implicit data type conversions

These tables show the outcome of expressions using the given operator and the shown data types as inputs. For example, using the + operator on two strings results in a string.

+ operator
INTEGER DOUBLE STRING BOOL
INTEGER INTEGER DOUBLE ERROR ERROR
DOUBLE DOUBLE DOUBLE ERROR ERROR
STRING ERROR ERROR STRING ERROR
BOOL ERROR ERROR ERROR ERROR

In the case of int or double addition, overflow can occur if the resulting value exceeds the available precision. During string concatenation, if the resulting string exceeds the allowed length of the string data type, the operation throws an error.

- operator
INTEGER DOUBLE STRING BOOL
INTEGER INTEGER DOUBLE ERROR ERROR
DOUBLE DOUBLE DOUBLE ERROR ERROR
STRING ERROR ERROR ERROR ERROR
BOOL ERROR ERROR ERROR ERROR
* operator
INTEGER DOUBLE STRING BOOL
INTEGER INTEGER DOUBLE ERROR ERROR
DOUBLE DOUBLE DOUBLE ERROR ERROR
STRING ERROR ERROR ERROR ERROR
BOOL ERROR ERROR ERROR ERROR
/ operator
INTEGER DOUBLE STRING BOOL
INTEGER DOUBLE DOUBLE ERROR ERROR
DOUBLE DOUBLE DOUBLE ERROR ERROR
STRING ERROR ERROR ERROR ERROR
BOOL ERROR ERROR ERROR ERROR
<, >, <=, >= comparison operators
INTEGER DOUBLE STRING BOOL
INTEGER BOOL BOOL ERROR ERROR
DOUBLE BOOL BOOL ERROR ERROR
STRING ERROR ERROR ERROR ERROR
BOOL ERROR ERROR ERROR ERROR
==, != comparison operators
INTEGER DOUBLE STRING BOOL NULL
INTEGER BOOL BOOL ERROR ERROR BOOL
DOUBLE BOOL BOOL ERROR ERROR BOOL
STRING ERROR ERROR BOOL ERROR BOOL
BOOL ERROR ERROR ERROR BOOL BOOL
NULL BOOL BOOL BOOL BOOL BOOL

Dictionaries

Workflows supports dictionaries that can hold a user-defined structure of variables or arrays.

Dictionary definition

To define a dictionary, add an assign step to the workflow:

- STEP_NAME_A:
    assign:
      - DICTIONARY_A:
          KEY_1: VALUE_1
          KEY_2:
              KEY_3: VALUE_2

For example:

- createDictionary:
    assign:
      - myDictionary:
          FirstName: "John"
          LastName: "Smith"
          Age: 26
          Address:
              Street: "Flower Road 12"
              City: "Superville"
              Country: "Atlantis"

Reading dictionary values

To read the values in a dictionary, use the following structure within an expression:

${DICTIONARY.KEY}

Use an additional period to access the value of a nested key. For example, to return the value of Country from the previous example:

- lastStep:
    return: ${myDictionary.Address.Country}

Checking existence of a key in a dictionary

To check whether a given key is present in a dictionary, use the following expression:

${KEY in DICTIONARY}

For example:

- MyStep:
    switch:
      - condition: ${"Age" in myDictionary}
        next: AgeExists

To check whether a key is not in a dictionary, use the not() function:

- MyStep:
    switch:
      - condition: ${not("Age" in myDictionary)}
        next: AgeMissing

Subworkflows

You can use subworkflows to define a piece of logic that can be called from the main workflow, similar to a routine or function in a programming language. They allow you to repeat steps in a workflow without duplicating the steps and increasing the number of lines in the workflow's definition. Subworkflows can accept input parameters and return values.

If a workflow has a subworkflow, the main workflow must be placed in a main block. Subworkflows are always defined after the main body of the workflow definition:

main:
    steps:
        - STEP_NAME:
            ...
        ...

SUBWORKFLOW_NAME:
    [params: [PARAMETER_1,PARAMETER_2...]]
    steps:
        - SUBWORKFLOW_STEP_NAME:
        ...

You call a subworkflow using the call field within a workflow step:

call: SUBWORKFLOW_NAME

This example defines a subworkflow and calls it from the main workflow:

main:
    steps:
        - call_subworkflow:
            call: name_message
            args:
                first_name: "Sherlock"
                last_name: "Holmes"
            result: output
        - return_message:
            return: ${output}

name_message:
    params: [first_name, last_name]
    steps:
        - prepMessage:
            return: ${"Hello " + first_name + " " + last_name}

Default paremeters in subworkflows

Subworkflows support default values for parameters. Default parameter value is used as only if the parameter wasn't provided as part of a subworkflow call.

SUBWORKFLOW_NAME:
    [params: [PARAMETER_1, PARAMETER_2: DEFAULT_VALUE2...]]
    steps:
        - SUBWORKFLOW_STEP_NAME:
        ...

For example, the following workflow can be called with or without Country parameter, and if the Country is not specified, it deefaults to "United States".

build_address:
    params: [Street, ZipCode, Country: "United States"]
    steps:
        - concatenate:
            return: ${Street + ", " + ZipCodee + ", " + Country}

For more information about working with subworkflows, see Creating and using subworkflows.

Error handling

Workflows can retry a step when it encounters an exception instead of failing and ending the execution attempt. The Workflows syntax has several types of built-in error handling strategies:

  • Raising exceptions
  • Catching and handling HTTP request errors
  • Retrying a failed step

Raising exceptions

Raising an exception stops the execution with a fail state and returns the exception as an error message. Exceptions can be either a string or a dictionary.

String

- step_a:
    raise: "Something went wrong."

Dictionary

- step_a:
    raise:
        code: 55
        message: "Something went wrong."

Catching and handling HTTP request errors

Workflows considers any HTTP request that returns status code 400 or above failed. This makes the workflow execution fail unless the workflow catches and handles the error. Workflows uses a try/except structure for error handling:

- STEP_NAME:
   try:
       call: http.get
       ...
   except:
       as: error_dictionary
       steps:
           ...

error_dictionary: Name of a dictionary variable that contains the error message. For HTTP requests, the error dictionary has the following attributes:

  • code: HTTP status code
  • message: Human-readable error message
  • tags: Error tags. Tags can be any of the following strings:
    • HttpError: HTTP response received with status code >= 400
    • ConnectionError: Error connecting to the API endpoint. For example, due to incorrect domain name, DNS resolution problem, etc.
    • TimeoutError: Timeout value reached without receiving a response

For example:

- read_item:
    try:
        call: http.get
        args:
            url: https://host.com/api
        result: api_response
    except:
        as: e
        steps:
            - known_errors:
                switch:
                - condition: ${not("HttpError" in e.tags)}
                  return: "Connection problem."
                - condition: ${e.code == 404}
                  return: "Sorry, URL wasn’t found."
                - condition: ${e.code == 403}
                  return: "Authentication error."
            - unhandled_exception:
                raise: ${e}
- url_found:
    return: ${api_response.body}

The try block can contain multiple steps, allowing them to share the same except block for error handling:

- read_item:
    try:
        steps:
            - step_a:
                call: http.get
                args:
                    url: https://host.com/api
                result: api_response1
            - step_b:
                call: http.get
                args:
                    url: https://host.com/api2
                result: api_response2
                ...
    except:
        as: e
        steps:
            - KnownErrors:
            ...

Retrying steps

You can retry steps that return a specific error code, for example, particular HTTP status codes. The retry syntax allows you to:

  • Define the maximum number of retry attempts
  • Define a backoff model to increase the likelihood of success

Workflows has default retry policies available for both idempotent and non-idempotent steps. Additionally, the default retry policies can be modified, and you can create a custom retry policy if the existing ones don't work for your use case. If you use a default retry policy, you don't need to specify a predicate or define the retry configuration values.

A retry policy is composed of a predicate that defines which error codes should be retried and default values for the retry configuration values.

- step_name:
    try:
        steps:
            ...
    retry: [${http.default_retry} | ${http.default_retry_non_idempotent}]
        [predicate: ${http.default_retry_predicate} | ${http.default_retry_predicate_non_idempotent}]
        [max_retries: number_of_retries]
        [backoff:
            initial_delay: delay_seconds
            max_delay: max_delay_seconds
            multiplier: delay_multiplier]
  • retry: Optional. If omitted, all other fields are required. Options include ${http.default_retry} and ${http.default_retry_non_idempotent}. Allows you to specify a default retry policy to use. If you specify a retry policy, omit all other fields in the retry block.
  • predicate: Required if you don't select a default retry policy. Defines which error codes will be retried. Options include ${http.default_retry_predicate}, ${http.default_retry_predicate_non_idempotent}, or a custom predicate defined as a subworkflow.
  • Retry configuration values: Required if a default retry policy is not used.

    • max_retries: Maximum number of times a step will be retried.
    • backoff: Block that controls how retries occur. Has the following parameters:

      • initial_delay: Delay in seconds between the initial failure and the first retry.
      • max_delay: Maximum delay in seconds between retries.
      • delay_multiplier: Multiplier applied to the previous delay to calculate the delay for the subsequent retry.

    For example, given the following retry configuration values:

            max_retries: 8
            backoff:
                initial_delay: 1
                max_delay: 60
                multiplier: 2
    

    The step will be retried a total of eight times. The initial delay is 1 second, and the delay is doubled on each attempt, with a maximum delay of 60 seconds. In this case, the delays between subsequent attempts are: 1, 2, 4, 8, 16, 32, 60, and 60 (time given in seconds). After eight attempts, the step is considered failed and an exception is raised.

You can configure the retry block in one of three ways:

Using a default retry policy

There are two default retry policies available: one for idempotent steps, and one for non-idempotent steps.

The default retry policies are composed of a default retry predicate and a set of default retry configuration values.

Default retry policy for idempotent steps

Note: You should only use this retry policy for idempotent steps (steps that can be safely repeated.)

The default retry policy for idempotent steps has the following configuration:

  • predicate: ${http.default_retry_predicate}. Retries HTTP status codes [429, 502, 503, 504], connection error, or timeout
  • max_retries: 5
  • initial_delay: 1.0
  • max_delay: 60
  • multiplier: 1.25

For example, to use the default retry policy for an idempotent step:

    - idempotent_step:
        try:
            call: http.get
            args:
                url: https://host.com/api
        retry: ${http.default_retry}
Default retry for non-idempotent steps

Note: You should only use this retry policy for non-idempotent steps (steps that can't be safely repeated.)

The default retry policy for non-idempotent steps has the following configuration:

  • predicate: ${http.default_retry_predicate_non_idempotent}. Retries HTTP status codes [429, 503]
  • max_retries: 5
  • initial_delay: 1.0
  • max_delay: 60
  • multiplier: 1.25

For example, to use the default retry policy for a non-idempotent step:

    - non_idempotent_step:
        try:
            call: http.get
            args:
                url: https://host.com/api
        retry: ${http.default_retry_non_idempotent}

Using a default retry predicate with custom retry configuration

Select the appropriate predicate for the type of step (idempotent or non-idempotent), then supply the desired retry configuration values. For example, the following retry policy uses the default predicate for non-idempotent steps and defines custom configuration values:

- step_name:
    try:
        steps:
            ...
    retry:
        predicate: ${http.default_retry_predicate_non_idempotent}
        max_retries: 10
        backoff:
            initial_delay: 1
            max_delay: 90
            multiplier: 3

Creating a custom retry policy

To create a custom retry policy, use a subworkflow to define your predicate (the set of errors the policy will be called for). Then call your predicate and define your desired retry configuration values in the retry block in the main workflow.

main:
    - step_name:
        try:
            steps:
                ...
        retry:
            predicate: ${retry_predicate}
            max_retries: number_of_retries
            backoff:
                initial_delay: delay_seconds
                max_delay: max_delay_seconds
                multiplier: delay_multiplier

retry_predicate:
    params: [e]
    steps:
        ...

For example, the following code implements a custom retry policy that only retries HTTP requests that returned HTTP status code 500:

main:
  steps:
    - read_item:
        try:
          call: http.get
          args:
            url: https://host.com/api
          result: api_response
        retry:
          predicate: ${custom_predicate}
          max_retries: 5
          backoff:
            initial_delay: 2
            max_delay: 60
            multiplier: 2
    - last_step:
        return: "OK"

custom_predicate:
    params: [e]
    steps:
      - what_to_repeat:
          switch:
          - condition: ${e.code == 500}
              return: True
      - otherwise:
          return: False

What's next