Migrate from AWS Step Functions to Workflows

To help you prepare to migrate from Amazon Web Services (AWS) Step Functions to Workflows on Google Cloud, this page explains key similarities and differences between the two products. This information is intended to assist those who are already familiar with Step Functions implement a similar architecture using Workflows.

Like Step Functions, Workflows is a fully-managed, state-based orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.

Note that Step Functions Express Workflows is an AWS Step Functions workflow type that is not considered here since the duration of an Express Workflow is limited, and an exactly-once workflow execution is not supported.

Hello world

In Step Functions, a state machine is a workflow, and a task is a state in a workflow that represents a single unit of work that another AWS service performs. Step Functions requires every state to define the next state.

In Workflows, a series of steps using the Workflows syntax describes the tasks to execute. Workflows treats steps as if they are in an ordered list and executes them one at a time until all the steps have run.

The following "Hello world" sample demonstrates the use of states in Step Functions and steps in Workflows:

Step Functions

  {
    "Comment": "Hello world example of Pass states in Amazon States Language",
    "StartAt": "Hello",
    "States": {
      "Hello": {
        "Type": "Pass",
        "Result": "Hello",
        "Next": "World"
      },
      "World": {
        "Type": "Pass",
        "Result": "World",
        "End": true
      }
    }
  }

Workflows YAML

  ---
  # Hello world example of steps using Google Cloud Workflows syntax
  main:
      steps:
      - Hello:
          next: World
      - World:
          next: end
  ...

Workflows JSON

  {
    "main": {
      "steps": [
        {
          "Hello": {
            "next": "World"
          }
        },
        {
          "World": {
            "next": "end"
          }
        }
      ]
    }
  }

Comparison overview

This section compares the two products in more detail.

Step FunctionsWorkflows
SyntaxJSON (YAML in tooling) YAML or JSON
Control flowTransition among states Imperative flow control with steps
WorkerResources (ARN) HTTP requests and connectors
ReliabilityCatch/retry Catch/retry
ParallelismSupported Available through experimental.executions.map
State dataState is passed along Workflows variables
AuthenticationIAM IAM
User experienceWorkflow Studio, CLI, SDK, IaC Google Cloud console, CLI, SDK, IaC
Pricing Step Functions pricing Workflows pricing
Syntax

Step Functions primarily uses JSON to define functions and does not support YAML directly; however, in the AWS Toolkit for Visual Studio Code and in AWS CloudFormation, you can use YAML for a Step Functions definition.

You can describe Workflows steps using the Workflows syntax, and they can be written in either YAML or JSON. The majority of workflows are in YAML. The examples on this page demonstrate the advantages of YAML, including ease of reading and writing, and native support of comments. For a detailed explanation of the Workflows syntax, see the Syntax reference.

Control flow

Both Workflows and Step Functions model workflows as a series of tasks: steps in Workflows and states in Step Functions. Both allow a task to indicate which task to execute next and support switch-like conditionals to pick the next unit of work based on the current state. One key difference is that Step Functions requires every state to define the next one, while Workflows executes steps in the order that they are specified (including alternative next steps). For more information, see Conditions and Steps.

Worker

Both products orchestrate compute resources such as functions, containers, and other web services to get things done. In Step Functions, the worker is identified by a Resource field which is syntactically a URI. The URIs used to identify worker resources are in Amazon Resource Name (ARN) format, and users cannot directly invoke an arbitrary HTTP endpoint.

Workflows can send HTTP requests to an arbitrary HTTP endpoint and get a response. Connectors make it easier to connect to other Google Cloud APIs within a workflow, and to integrate your workflows with other Google Cloud products like Pub/Sub, BigQuery, or Cloud Build. Connectors simplify calling services because they handle the formatting of requests for you, providing methods and arguments so that you don't need to know the details of a Google Cloud API. You can also configure the timeout and polling policies.

Reliability

If a task fails, your workflow must be able to retry appropriately, catch exceptions when it has to, and reroute the workflow as necessary. Both Step Functions and Workflows achieve reliability using similar mechanisms: exception catching with retries, and eventual dispatching to elsewhere in the workflow. For more information, see Workflow errors.

Parallelism

You might want your workflow to orchestrate multiple tasks in parallel. Step Functions provides two ways to achieve this: you can take one data item and pass it in parallel to multiple different tasks; or, you can use an array and pass its elements to the same task.

You can use a Workflows experimental feature, experimental.executions.map, to support parallel work.

State data

One benefit of a workflow engine is that state data is maintained for you without an external datastore. In Step Functions, the state data is passed along in a JSON structure from one state to another.

In Workflows, you can save the state data in global variables. Since you are allowed up to a one-year execution duration, you can keep the state data as long as the instance is still in execution.

Authentication

Both products rely upon an underlying Identity and Access Management (IAM) system for authentication and access control. For example, you can use an IAM role to invoke Step Functions.

In Workflows, you can use a service account to invoke a workflow; you can use OAuth 2.0 or OIDC to connect with Google Cloud APIs; and you can use an authorization request header to authenticate with a third-party API. For more information, see Grant a workflow permission to access Google Cloud resources and Make authenticated requests from a workflow.

User experience

You can use a command-line tool, or infrastructure as code (IaC) such as Terraform, to define and manage both Step Functions and Workflows.

In addition, Workflows supports the execution of workflows using the client libraries, in the Cloud console, using the Google Cloud CLI, or by sending a request to the Workflows REST API. For details, see Execute a workflow.

Pricing

Both products have a free tier. For more details, see their respective pricing pages: Step Functions pricing and Workflows pricing.

Mapping state types to steps

There are eight state types in Step Functions. States are elements in a state machine that can make decisions based on their input, perform actions, and pass output to other states. Before migrating from Step Functions to Workflows, ensure that you understand how to translate each state type to a Workflows step.

Choice

A Choice state adds branching logic to a state machine.

In Workflows, you can use a switch block as a selection mechanism that allows the value of an expression to control the flow of a workflow's execution. If a value matches, that condition's statement is executed. For more information, see Conditions.

Step Functions

  "ChoiceState": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.foo",
        "NumericEquals": 1,
        "Next": "FirstMatchState"
      },
      {
        "Variable": "$.foo",
        "NumericEquals": 2,
        "Next": "SecondMatchState"
      }
    ],
    "Default": "DefaultState"
  }

Workflows YAML

  switch:
    - condition: ${result.body.SomeField < 10}
      next: small
    - condition: ${result.body.SomeField < 100}
      next: medium
    - condition: true
      next: default_step

Workflows JSON

  {
    "switch": [
      {
        "condition": "${result.body.SomeField < 10}",
        "next": "small"
      },
      {
        "condition": "${result.body.SomeField < 100}",
        "next": "medium"
      },
      {
        "condition": true,
        "next": "default_step"
      }
    ]
  }

Fail

A Fail state stops the execution of the state machine and marks it as a failure.

In Workflows, you can raise custom errors using the raise syntax, and you can catch and handle errors using a try/except block. For more information, see Raise errors.

Step Functions

  "FailState": {
      "Type": "Fail",
      "Error": "ErrorA",
      "Cause": "Kaiju attack"
  }

Workflows YAML

  raise:
      code: 55
      message: "Something went wrong."

Workflows JSON

  {
    "raise": {
      "code": 55,
      "message": "Something went wrong."
    }
  }

Map

A Map state can be used to run a set of steps for each element of an input array.

In Workflows, you can use a for loops for iterations.

Step Functions

  { "StartAt": "ExampleMapState",
    "States": {
      "ExampleMapState": {
        "Type": "Map",
        "Iterator": {
           "StartAt": "CallLambda",
           "States": {
             "CallLambda": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
               "End": true
             }
           }
        }, "End": true
      }
    }
  }

Workflows YAML

  - assignStep:
      assign:
        - map:
            1: 10
            2: 20
            3: 30
        - sum: 0
  - loopStep:
      for:
          value: key
          in: ${keys(map)}
          steps:
            - sumStep:
                assign:
                  - sum: ${sum + map[key]}
  - returnStep:
      return: ${sum}

Workflows JSON

  [
    {
      "assignStep": {
        "assign": [
          {
            "map": {
              "1": 10,
              "2": 20,
              "3": 30
            }
          },
          {
            "sum": 0
          }
        ]
      }
    },
    {
      "loopStep": {
        "for": {
          "value": "key",
          "in": "${keys(map)}",
          "steps": [
            {
              "sumStep": {
                "assign": [
                  {
                    "sum": "${sum + map[key]}"
                  }
                ]
              }
            }
          ]
        }
      }
    },
    {
      "returnStep": {
        "return": "${sum}"
      }
    }
  ]

Parallel

A Parallel state can be used to create parallel branches of execution in your state machine. In the following Step Functions example, an address and phone lookup is performed in parallel.

In Workflows, you can use an experimental feature, experimental.executions.map, to support parallel work.

Step Functions

  { "StartAt": "LookupCustomerInfo",
    "States": {
      "LookupCustomerInfo": {
        "Type": "Parallel",
        "End": true,
        "Branches": [
          {
           "StartAt": "LookupAddress",
           "States": {
             "LookupAddress": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:AddressFinder",
               "End": true
             }
           }
         },
         {
           "StartAt": "LookupPhone",
           "States": {
             "LookupPhone": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:PhoneFinder",
               "End": true
             }
           }
         }
        ]
      }
    }
  }

Workflows YAML

  - parallel-executor:
      call: experimental.executions.map
      args:
        workflow_id: LookupCustomerInfo
        arguments: [{"LookupAddress":"${cust_id}"},{"LookupPhone":"${cust_id}"}]
      result: result

Workflows JSON

  [
    {
      "parallel-executor": {
        "call": "experimental.executions.map",
        "args": {
          "workflow_id": "LookupCustomerInfo",
          "arguments": [
            {
              "LookupAddress": "${cust_id}"
            },
            {
              "LookupPhone": "${cust_id}"
            }
          ]
        },
        "result": "result"
      }
    }
  ]

Pass

A Pass state passes its input to its output, without performing work. This is commonly used to manipulate the state data in the JSON.

As Workflows does not pass along data in this way, you can leave the state as a no-op, or you can use an assign step to modify the variables. For more information, see Assign variables.

Step Functions

  "No-op": {
    "Type": "Pass",
    "Result": {
      "x-datum": 0.38,
      "y-datum": 622.22
    },
    "ResultPath": "$.coords",
    "Next": "End"
  }

Workflows YAML

  assign:
      - number: 5
      - number_plus_one: ${number+1}
      - other_number: 10
      - string: "hello"

Workflows JSON

  {
    "assign": [
      {
        "number": 5
      },
      {
        "number_plus_one": "${number+1}"
      },
      {
        "other_number": 10
      },
      {
        "string": "hello"
      }
    ]
  }

Succeed

A Succeed state stops an execution successfully.

In Workflows, you can use return in the main workflow to stop a workflow's execution. You can also finish a workflow by completing the final step (assuming that the step doesn't jump to another), or you can use next: end to stop a workflow's execution if you don't need to return a value. For more information, see Complete the execution of a workflow.

Step Functions

  "SuccessState": {
    "Type": "Succeed"
  }

Workflows YAML

  return: "Success!"
  next: end

Workflows JSON

  {
    "return": "Success!",
    "next": "end"
  }

Task

A Task state represents a single unit of work performed by a state machine. In the following Step Functions example, it invokes a Lambda function. (Activities are an AWS Step Functions feature that enables you to have a task in your state machine where the work is performed elsewhere.)

In the Workflows example, a call is made to an HTTP endpoint to invoke a Cloud Function. You can also use a connector which allows easy access to other Google Cloud products. Additionally, you can pause a workflow and poll for data. Or, you can use a callback endpoint to signal to your workflow that a specified event has occurred, and wait on that event without polling.

Step Functions

  "HelloWorld": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
    "End": true
  }

Workflows YAML

  - HelloWorld:
      call: http.get
      args:
          url: https://REGION-PROJECT_ID.cloudfunctions.net/helloworld
      result: helloworld_result

Workflows JSON

  [
    {
      "HelloWorld": {
        "call": "http.get",
        "args": {
          "url": "https://REGION-PROJECT_ID.cloudfunctions.net/helloworld"
        },
        "result": "helloworld_result"
      }
    }
  ]

Wait

A Wait state delays the state machine from continuing for a specified time.

You can use the Workflows sys.sleep standard library function to suspend execution for the given number of seconds to a maximum of 31536000 (one year).

Step Functions

  "wait_ten_seconds" : {
    "Type" : "Wait",
    "Seconds" : 10,
    "Next": "NextState"
  }

Workflows YAML

  - someSleep:
      call: sys.sleep
      args:
          seconds: 10

Workflows JSON

  [
    {
      "someSleep": {
        "call": "sys.sleep",
        "args": {
          "seconds": 10
        }
      }
    }
  ]

Example: Orchestrate microservices

The following Step Functions example checks a stock price, determines whether to buy or sell, and reports the result. The state machine in the sample integrates with AWS Lambda by passing parameters, uses an Amazon SQS queue to request human approval, and uses an Amazon SNS topic to return the results of the query.

{
      "StartAt": "Check Stock Price",
      "Comment": "An example of integrating Lambda functions in Step Functions state machine",
      "States": {
          "Check Stock Price": {
              "Type": "Task",
              "Resource": "CHECK_STOCK_PRICE_LAMBDA_ARN",
              "Next": "Generate Buy/Sell recommendation"
          },
          "Generate Buy/Sell recommendation": {
              "Type": "Task",
              "Resource": "GENERATE_BUY_SELL_RECOMMENDATION_LAMBDA_ARN",
              "ResultPath": "$.recommended_type",
              "Next": "Request Human Approval"
          },
          "Request Human Approval": {
              "Type": "Task",
              "Resource": "arn:PARTITION:states:::sqs:sendMessage.waitForTaskToken",
              "Parameters": {
                  "QueueUrl": "REQUEST_HUMAN_APPROVAL_SQS_URL",
                  "MessageBody": {
                      "Input.$": "$",
                      "TaskToken.$": "$$.Task.Token"
                  }
              },
              "ResultPath": null,
              "Next": "Buy or Sell?"
          },
          "Buy or Sell?": {
              "Type": "Choice",
              "Choices": [
                  {
                      "Variable": "$.recommended_type",
                      "StringEquals": "buy",
                      "Next": "Buy Stock"
                  },
                  {
                      "Variable": "$.recommended_type",
                      "StringEquals": "sell",
                      "Next": "Sell Stock"
                  }
              ]
          },
          "Buy Stock": {
              "Type": "Task",
              "Resource": "BUY_STOCK_LAMBDA_ARN",
              "Next": "Report Result"
          },
          "Sell Stock": {
              "Type": "Task",
              "Resource": "SELL_STOCK_LAMBDA_ARN",
              "Next": "Report Result"
          },
          "Report Result": {
              "Type": "Task",
              "Resource": "arn:PARTITION:states:::sns:publish",
              "Parameters": {
                  "TopicArn": "REPORT_RESULT_SNS_TOPIC_ARN",
                  "Message": {
                      "Input.$": "$"
                  }
              },
              "End": true
          }
      }
  }

Migrate to Workflows

To migrate the preceding Step Functions example to Workflows, you can create the equivalent Workflows steps by integrating Cloud Functions, supporting a callback endpoint that waits for HTTP requests to arrive at that endpoint, and using a Workflows connector to publish to a Pub/Sub topic in place of the Amazon SNS topic:

  1. Complete the steps to create a workflow but do not deploy it yet.

  2. In the workflow's definition, add a step to create a callback endpoint that waits for human input, and a step that uses a Workflows connector to publish to a Pub/Sub topic. For example:

    Workflows YAML

      ---
      main:
        steps:
          - init:
              assign:
                - projectId: '${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}'
                - region: LOCATION
                - topic: PUBSUB_TOPIC_NAME
          - Check Stock Price:
              call: http.get
              args:
                url: ${"https://" + region + "-" + projectId + ".cloudfunctions.net/CheckStockPrice"}
                auth:
                  type: OIDC
              result: stockPriceResponse
          - Generate Buy/Sell Recommendation:
              call: http.get
              args:
                url: ${"https://" + region + "-" + projectId + ".cloudfunctions.net/BuySellRecommend"}
                auth:
                  type: OIDC
                query:
                  price: ${stockPriceResponse.body.stock_price}
              result: recommendResponse
          - Create Approval Callback:
              call: events.create_callback_endpoint
              args:
                  http_callback_method: "GET"
              result: callback_details
          - Print Approval Callback Details:
              call: sys.log
              args:
                  severity: "INFO"
                  text: ${"Listening for callbacks on " + callback_details.url}
          - Await Approval Callback:
              call: events.await_callback
              args:
                  callback: ${callback_details}
                  timeout: 3600
              result: approvalResponse
          - Approval?:
              try:
                switch:
                  - condition: ${approvalResponse.http_request.query.response[0] == "yes"}
                    next: Buy or Sell?
              except:
                as: e
                steps:
                  - unknown_response:
                      raise: ${"Unknown response:" + e.message}
                      next: end
          - Buy or Sell?:
              switch:
                - condition: ${recommendResponse.body == "buy"}
                  next: Buy Stock
                - condition: ${recommendResponse.body == "sell"}
                  next: Sell Stock
                - condition: true
                  raise: ${"Unknown recommendation:" + recommendResponse.body}
          - Buy Stock:
              call: http.post
              args:
                url: ${"https://" + region + "-" + projectId + ".cloudfunctions.net/BuyStock"}
                auth:
                  type: OIDC
                body:
                  action: ${recommendResponse.body}
              result: message
          - Sell Stock:
              call: http.post
              args:
                url: ${"https://" + region + "-" + projectId + ".cloudfunctions.net/SellStock"}
                auth:
                  type: OIDC
                body:
                  action: ${recommendResponse.body}
              result: message
          - Report Result:
              call: googleapis.pubsub.v1.projects.topics.publish
              args:
                topic: ${"projects/" + projectId + "/topics/" + topic}
                body:
                  messages:
                  - data: '${base64.encode(json.encode(message))}'
              next: end
      ...

    Workflows JSON

      {
        "main": {
          "steps": [
            {
              "init": {
                "assign": [
                  {
                    "projectId": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"
                  },
                  {
                    "region": "LOCATION"
                  },
                  {
                    "topic": [
                      "PUBSUB_TOPIC_NAME"
                    ]
                  }
                ]
              }
            },
            {
              "Check Stock Price": {
                "call": "http.get",
                "args": {
                  "url": "${\"https://\" + region + \"-\" + projectId + \".cloudfunctions.net/CheckStockPrice\"}",
                  "auth": {
                    "type": "OIDC"
                  }
                },
                "result": "stockPriceResponse"
              }
            },
            {
              "Generate Buy/Sell Recommendation": {
                "call": "http.get",
                "args": {
                  "url": "${\"https://\" + region + \"-\" + projectId + \".cloudfunctions.net/BuySellRecommend\"}",
                  "auth": {
                    "type": "OIDC"
                  },
                  "query": {
                    "price": "${stockPriceResponse.body.stock_price}"
                  }
                },
                "result": "recommendResponse"
              }
            },
            {
              "Create Approval Callback": {
                "call": "events.create_callback_endpoint",
                "args": {
                  "http_callback_method": "GET"
                },
                "result": "callback_details"
              }
            },
            {
              "Print Approval Callback Details": {
                "call": "sys.log",
                "args": {
                  "severity": "INFO",
                  "text": "${\"Listening for callbacks on \" + callback_details.url}"
                }
              }
            },
            {
              "Await Approval Callback": {
                "call": "events.await_callback",
                "args": {
                  "callback": "${callback_details}",
                  "timeout": 3600
                },
                "result": "approvalResponse"
              }
            },
            {
              "Approval?": {
                "try": {
                  "switch": [
                    {
                      "condition": "${approvalResponse.http_request.query.response[0] == \"yes\"}",
                      "next": "Buy or Sell?"
                    }
                  ]
                },
                "except": {
                  "as": "e",
                  "steps": [
                    {
                      "unknown_response": {
                        "raise": "${\"Unknown response:\" + e.message}",
                        "next": "end"
                      }
                    }
                  ]
                }
              }
            },
            {
              "Buy or Sell?": {
                "switch": [
                  {
                    "condition": "${recommendResponse.body == \"buy\"}",
                    "next": "Buy Stock"
                  },
                  {
                    "condition": "${recommendResponse.body == \"sell\"}",
                    "next": "Sell Stock"
                  },
                  {
                    "condition": true,
                    "raise": "${\"Unknown recommendation:\" + recommendResponse.body}"
                  }
                ]
              }
            },
            {
              "Buy Stock": {
                "call": "http.post",
                "args": {
                  "url": "${\"https://\" + region + \"-\" + projectId + \".cloudfunctions.net/BuyStock\"}",
                  "auth": {
                    "type": "OIDC"
                  },
                  "body": {
                    "action": "${recommendResponse.body}"
                  }
                },
                "result": "message"
              }
            },
            {
              "Sell Stock": {
                "call": "http.post",
                "args": {
                  "url": "${\"https://\" + region + \"-\" + projectId + \".cloudfunctions.net/SellStock\"}",
                  "auth": {
                    "type": "OIDC"
                  },
                  "body": {
                    "action": "${recommendResponse.body}"
                  }
                },
                "result": "message"
              }
            },
            {
              "Report Result": {
                "call": "googleapis.pubsub.v1.projects.topics.publish",
                "args": {
                  "topic": "${\"projects/\" + projectId + \"/topics/\" + topic}",
                  "body": {
                    "messages": [
                      {
                        "data": "${base64.encode(json.encode(message))}"
                      }
                    ]
                  }
                },
                "next": "end"
              }
            }
          ]
        }
      }

    Replace the following:

    • LOCATION: a supported Google Cloud region. For example, us-central1.
    • PUBSUB_TOPIC_NAME: the name of your Pub/Sub topic. For example, my_stock_example.
  3. Deploy and then execute the workflow.

  4. During the workflow's execution, it pauses and waits for you to invoke the callback endpoint. You can use a curl command to do this. For example:

    curl -H "Authorization: Bearer $(gcloud auth print-access-token)"
    https://workflowexecutions.googleapis.com/v1/projects/CALLBACK_URL?response=yes
    

    Replace CALLBACK_URL with the rest of the path to your callback endpoint.

  5. Once the workflow has successfully completed, you can receive the message from the Pub/Sub subscription. For example:

    gcloud pubsub subscriptions pull stock_example-sub  --format="value(message.data)" | jq
    

    The output message should be similar to the following (either a buy or a sell):

    {
      "body": "buy",
      "code": 200,
      "headers": {
      [...]
      }
    

What's next