Create and manage workflows

You can create and manage workflows in the Google Cloud console, or by using the Google Cloud CLI in either your terminal or Cloud Shell. You can also manage workflows through the Workflows API.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

Console

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Workflows API.

    Enable the API

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Workflows API.

    Enable the API

gcloud

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. Install the Google Cloud CLI.
  3. To initialize the gcloud CLI, run the following command:

    gcloud init
  4. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  5. Make sure that billing is enabled for your Google Cloud project.

  6. Enable the Workflows API:

    gcloud services enable workflows.googleapis.com
  7. Install the Google Cloud CLI.
  8. To initialize the gcloud CLI, run the following command:

    gcloud init
  9. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  10. Make sure that billing is enabled for your Google Cloud project.

  11. Enable the Workflows API:

    gcloud services enable workflows.googleapis.com

REST

To manage workflows using the Workflows API, we recommend that you use the Google-provided client libraries to call the workflows.googleapis.com service. For more information, see Workflows API.

Create a service account

A service account represents the identity of a workflow and determines what permissions the workflow has and which Google Cloud resources it can access. Create a service account if you don't already have one; then grant it the roles necessary for managing workflows and creating logs.

If you don't specify a service account during the workflow's creation, the workflow uses the default Compute Engine service account for its identity. For more information, see Grant a workflow permission to access Google Cloud resources.

We strongly recommend using a service account with the least privileges necessary to access the required resources.

Note that to create a resource and attach a service account, you need permissions to create that resource and to impersonate the service account that you will attach to the resource. For more information, see Service account permissions.

Console

  1. In the Google Cloud console, go to the Service Accounts page.

    Go to Service Accounts

  2. Select a project and then click Create service account.

  3. In the Service account name field, enter a name.

    The name must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you can't change its name.

  4. Click Create and continue.

  5. Click Select a role.

  6. Select the following roles, clicking Add another role as required:

    1. To create, update, and execute workflows, select Workflows > Workflows Editor.
    2. To send logs to Cloud Logging, select Logging > Logs Writer.
  7. Click Done to finish creating the service account.

gcloud

  1. Create the service account.

    gcloud iam service-accounts create SERVICE_ACCOUNT
  2. Grant permissions to the service account by assigning roles.

    1. To create, update, and execute workflows, assign the roles/workflows.editor role:

      gcloud projects add-iam-policy-binding PROJECT_ID \
          --member "serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com" \
          --role "roles/workflows.editor"
      
    2. To send logs to Cloud Logging, assign the roles/logging.logWriter role:

      gcloud projects add-iam-policy-binding PROJECT_ID \
          --member "serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com" \
          --role "roles/logging.logWriter"
      

    Replace the following:

    • SERVICE_ACCOUNT: the name of the service account. It must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you can't change its name.

    • PROJECT_ID: your project's ID.

REST

You can use the serviceAccounts.create method to create a service account. For more information, see Create service accounts.

You can programmatically grant multiple roles by modifying and setting the allow policy for a resource using the setIamPolicy method. For more information, see Grant or revoke multiple roles programmatically.

Create a workflow

A workflow definition is made up of a series of steps described using the Workflows syntax, which can be written in either YAML or JSON format. After defining a workflow, you deploy it to make it available for execution. The deploy step also validates that the source file can be executed. It fails if the source file doesn't contain a valid workflow definition.

Editing YAML files can be error-prone. You can create your workflow by using a preferred IDE or source code editor and setting up autocompletion and syntax validation to reduce errors.

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

  2. Click Create.

  3. Enter a name for the workflow, such as myFirstWorkflow. The name can contain letters, numbers, underscores, and hyphens. It must start with a letter and end with a number or letter.

  4. Optionally, add a description of the workflow.

  5. In the Region list, select an appropriate location to deploy your workflow in; for example, us-central1.

  6. In the Service account list, select a service account that your workflow will use to access other Google Cloud services. For more information, in this document, see Create a service account.

  7. Optionally, do any of the following:

    1. Specify the level of call logging that you want to apply to the workflow definition. In the Call log level list, select one of:

      • Not specified: no logging level is specified. This is the default. An execution log level takes precedence over any workflow log level, unless the execution log level is not specified (the default); in that case, the workflow log level applies.
      • Errors only: log all caught exceptions; or when a call is stopped due to an exception.
      • All calls: log all calls to subworkflows or library functions and their results.
      • No logs: no call logging.
    2. Specify a Cloud Key Management Service key that the workflow should use for data encryption: select Customer-managed encryption key (CMEK). For more information, see Use customer-managed encryption keys.

    3. Specify an environment variable that is accessible by your workflow at runtime. For more information, see Use environment variables.

    4. Add a label: labels are key-value pairs that help you organize your Google Cloud instances. For more information, see What are labels? Workflow labels are inherited by executions of the workflow. To view execution labels, use the workflows.executions.list method to list and filter workflow executions.

    5. Schedule your workflow: select Add new trigger > Cloud Scheduler. For more information, see Schedule a workflow using Cloud Scheduler.

    6. Trigger your workflow through an event or Pub/Sub message: select Add new trigger > Eventarc. For more information, see Trigger a workflow with events or Pub/Sub messages.

  8. Click Next.

  9. In the workflow editor, enter the definition for your workflow. See an example workflow.

  10. Click Deploy.

gcloud

  1. Make sure your workflow's source code is saved in a YAML or JSON file, such as MY_WORKFLOW.YAML or MY_WORKFLOW.JSON. See an example workflow.

  2. Open a terminal.

  3. You can create and deploy a workflow using the gcloud workflows deploy command:

    gcloud workflows deploy WORKFLOW_NAME \
        --location=LOCATION \
        --call-log-level=CALL_LOGGING_LEVEL \
        --description="DESCRIPTION" \
        --labels="LABEL_KEY=LABEL_VALUE" \
        --set-env-vars="ENV_KEY=ENV_VALUE" \
        --kms-key=ENCRYPT_KEY \
        --service-account=SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com \
        --source=YAML_OR_JSON_SOURCE_FILE
    

    Replace the following:

    • WORKFLOW_NAME: the name of your workflow—for example, myFirstWorkflow. The name can contain letters, numbers, underscores, and hyphens. It must start with a letter and end with a number or letter.

    • LOCATION: the region to deploy your workflow in; for example, us-central1.

    • CALL_LOGGING_LEVEL: optional. Level of call logging to apply during execution. Can be one of:

      • none: no logging level is specified. This is the default. An execution log level takes precedence over any workflow log level, unless the execution log level is not specified (the default); in that case, the workflow log level applies.
      • log-errors-only: log all caught exceptions; or when a call is stopped due to an exception.
      • log-all-calls: log all calls to subworkflows or library functions and their results.
      • log-none: no call logging.
    • DESCRIPTION: optional. A description of the workflow.

    • LABEL_KEY=LABEL_VALUE: optional. List of label key-value pairs that helps you organize your Google Cloud instances; for example, name=wrench. For more information, see What are labels? Workflow labels are inherited by executions of the workflow. To view execution labels, use the workflows.executions.list method to list and filter workflow executions.

    • ENV_KEY=ENV_VALUE: optional. List of environment variable key-value pairs; for example, MONTH=January. For more information, see Use environment variables.

    • ENCRYPT_KEY: optional. A Cloud KMS key that the workflow should use for data encryption in the format projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME. For more information, see Use customer-managed encryption keys.

    • SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com: optional. The service account your workflow will use to access other Google Cloud services. For more information, in this document, see Create a service account.

    • YAML_OR_JSON_SOURCE_FILE: the source file for the workflow definition. For example: myFirstWorkflow.yaml.

REST

To create a new workflow with a specified name, use the projects.locations.workflows.create method, and use the workflowId query parameter to specify an ID for the workflow.

Before using any of the request data, make the following replacements:

  • WORKFLOW_NAME: the name of your workflow—for example, myFirstWorkflow. The name can contain letters, numbers, underscores, and hyphens. It must start with a letter and end with a number or letter.
  • DESCRIPTION: optional. A description of your workflow. Must be no longer than 1000 Unicode characters.
  • LABEL_KEY and LABEL_VALUE: optional. A map of label key and value pairs that help you organize your Google Cloud instances. For example: {"name": "wrench", "mass": "1kg", "count": "3"} For more information, see What are labels? Workflow labels are inherited by executions of the workflow. To view execution labels, use the workflows.executions.list method to list and filter workflow executions.
  • SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com: optional. The service account your workflow will use to access other Google Cloud services. The project ID is your Google Cloud project ID. For more information, in this document, see Create a service account.
  • ENCRYPT_KEY: optional. A Cloud KMS key that the workflow should use for data encryption in the format projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME. For more information, see Use customer-managed encryption keys.
  • CALL_LOGGING_LEVEL: optional. The call logging level to apply during execution. The default is that no logging level is specified and the workflow log level applies instead. For more information, see Send Logs to Logging. One of the following:
    • CALL_LOG_LEVEL_UNSPECIFIED: no logging level is specified and the workflow log level applies instead. This is the default. Otherwise, the execution log level applies and takes precedence over the workflow log level.
    • LOG_ERRORS_ONLY: log all caught exceptions; or when a call is stopped due to an exception.
    • LOG_ALL_CALLS: log all calls to subworkflows or library functions and their results.
    • LOG_NONE: no call logging.
  • ENV_KEY and ENV_VALUE: optional. A map of environment variable key and value pairs; for example, { "month": "January", "day": "Monday"}. For more information, see Use environment variables.
  • SOURCE_CODE: the workflow definition. You must escape new lines in YAML. For example: main:\n params:\n - input\n steps:\n - returnOutput:\n return: Hello

    You must escape quotation marks in JSON. For example: {\"main\":{\"params\":[\"input\"],\"steps\":[{\"returnOutput\":{\"return\":\"Hello\"}}]}}

  • LOCATION: the region in which the workflow is to be deployed—for example, us-central1.

Request JSON body:

{
  "name": "WORKFLOW_NAME",
  "description": "DESCRIPTION",
  "labels": {"LABEL_KEY":"LABEL_VALUE"},
  "serviceAccount": "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com",
  "cryptoKeyName": "ENCRYPT_KEY",
  "callLogLevel": "CALL_LOGGING_LEVEL",
  "userEnvVars": {"ENV_KEY":"ENV_VALUE"},
  "sourceContents": "SOURCE_CODE"
}

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.workflows.v1.OperationMetadata",
    "createTime": "2023-12-05T14:06:06.338390918Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/workflows/WORKFLOW_NAME",
    "verb": "create",
    "apiVersion": "v1"
  },
  "done": false
}

If the value of "done" is false, the operation is still in progress.

List workflows

You can list workflows or use filters to retrieve a specific workflow.

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

    This page lists your workflows in all locations, and includes details such as names, regions, latest revisions, and more.

  2. To filter your workflows:

    1. Click Filter or the Filter workflows field.
    2. In the Properties list, select an option to filter the workflows by.

    You can select a single property or use the logical operator OR to add more properties.

  3. To sort your workflows, beside any supported column heading, click Sort.

gcloud

List workflows using the gcloud workflows list command:

gcloud workflows list --location=LOCATION

Replace LOCATION with the ID or fully qualified identifier for the location of your workflow.

This command lists your workflows in the specified location, and includes details such as a workflow's NAME, STATE, REVISION_ID, and UPDATE_TIME.

REST

To list workflows in a given project and location, use the projects.locations.workflows.list method.

Alternatively, to retrieve the details of a single workflow, use the projects.locations.workflows.get method.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: the region in which the workflows are deployed—for example, us-central1.

To send your request, expand one of these options:

If successful, the response body contains instances of Workflow and the response should be similar to the following:

{
  "workflows": [
    {
      "name": "projects/PROJECT_ID/locations/LOCATION/workflows/WORKFLOW_NAME",
      "state": "ACTIVE",
      "revisionId": "000001-0ce",
      "createTime": "2023-12-08T13:56:59.306770745Z",
      "updateTime": "2023-12-08T13:56:59.547021939Z",
      "revisionCreateTime": "2023-12-08T13:56:59.340161044Z",
      "serviceAccount": "projects/PROJECT_ID/serviceAccounts/PROJECT_NUMBER-compute@developer.gserviceaccount.com",
      "sourceContents": "main:\n    params: [input]\n [...] return: '${wikiResult.body[1]}'\n"
    },
    {
      object (Workflow)
    }
  ],
  "nextPageToken": string,
  "unreachable": [
    string
  ]
}

Update a workflow

You can update an existing workflow. You must specify the name of the workflow you want to update and its source. Note that you can't change a workflow's name or location.

Updating a workflow does not affect in-progress executions. Only future executions of the workflow will use the updated configuration.

Each time you update a workflow, its versionID is updated. The versionID consists of two parts, separated by a hyphen:

  • A number, starting at one, that increments each time you update a workflow.

  • A random, three-character, alphanumeric string.

For example, 000001-27f indicates the initial version of a workflow and 000002-d52 indicates a workflow that has been updated once.

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

  2. Click the name of the workflow you want to update. Note that you can't change the workflow's name.

    The Workflow details page appears.

  3. You can edit the workflow in the following ways:

    • To edit the source:

      1. Click the Source tab.
      2. Click Edit.
      3. To save your changes, click Save. The updated workflow is deployed.
    • To update the description, the service account the workflow uses for authentication, the call log level, environment variables, labels, or the encryption key:

      1. Click the Details tab.
      2. Click the appropriate icon.
      3. If you are updating the call log level, select one of:
        • Not specified: no logging level is specified. This is the default. An execution log level takes precedence over any workflow log level, unless the execution log level is not specified (the default); in that case, the workflow log level applies.
        • Errors only: log all caught exceptions; or when a call is stopped due to an exception.
        • All calls: log all calls to subworkflows or library functions and their results.
        • No logs: no call logging.
      4. If you update an existing label, or add a new one, it can take up to 10 minutes for the new label to take effect. Workflow labels are inherited by executions of the workflow. To view execution labels, use the workflows.executions.list method to list and filter workflow executions.
      5. To save your changes, click Save. The updated workflow is deployed.
    • To edit the previous fields at the same time, or add or update a trigger:

      1. Click Edit.
      2. To edit the source, click Next.
      3. To save your changes and deploy the updated workflow, click Deploy.
  4. If you want to update the roles for your service account, click the Permissions tab.

    1. Principals are users, groups, domains, or service accounts. To update an existing principal:

      1. Find a row containing the principal.
      2. Click Edit principal in that row.
      3. Click Add another role or click Delete role.
    2. If you are adding a role, in the Select a role list, select an appropriate role.

    3. To add another role, click Add another role.

    4. Click Save.

gcloud

  1. Open a terminal.

  2. Find the name of the workflow you want to update. If you don't know the workflow's name, you can enter the following command to list all your workflows:

    gcloud workflows list
    
  3. Locate the YAML or JSON file where your workflow source is saved, such as WORKFLOW_NAME.YAML or WORKFLOW_NAME.JSON.

  4. You can update an existing workflow to change its source, description, labels, environment variables, the call log level, encryption key, or its associated service account using the gcloud workflows deploy command.

    You must specify the name of the workflow you want to update and its source; however, the rest of the flags are optional. To remove a customer-managed encryption key, use the --clear-kms-key flag.

    gcloud workflows deploy WORKFLOW_NAME \
        --call-log-level=CALL_LOGGING_LEVEL \
        --description="DESCRIPTION" \
        --labels="LABEL_KEY=LABEL_VALUE" \
        --set-env-vars="ENV_KEY=ENV_VALUE" \
        --kms-key=ENCRYPT_KEY \
        --service-account=SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com \
        --source=YAML_OR_JSON_SOURCE_FILE
    

    Replace the following:

    • WORKFLOW_NAME: required. The name of your workflow.

    • CALL_LOGGING_LEVEL: optional. Level of call logging to apply to the workflow. Can be one of:

      • none: no logging level is specified. This is the default. An execution log level takes precedence over any workflow log level, unless the execution log level is not specified (the default); in that case, the workflow log level applies.
      • log-errors-only: log all caught exceptions; or when a call is stopped due to an exception.
      • log-all-calls: log all calls to subworkflows or library functions and their results.
      • log-none: no call logging.
    • DESCRIPTION: optional. A description of the workflow.

    • LABEL_KEY=LABEL_VALUE: optional. List of label key-value pairs that helps you organize your Google Cloud instances; for example, name=wrench. For more information, see What are labels? Workflow labels are inherited by executions of the workflow. To view execution labels, use the workflows.executions.list method to list and filter workflow executions.

    • ENV_KEY=ENV_VALUE: optional. List of environment variable key-value pairs; for example, MONTH=January. For more information, see Use environment variables.

    • ENCRYPT_KEY: optional. A Cloud KMS key that the workflow should use for data encryption in the format projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME. For more information, see Use customer-managed encryption keys.

    • SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com: optional. The service account your workflow will use to access other Google Cloud services. If you want to update the roles for your service account, see Grant a workflow permission to access Google Cloud resources and Manage access to projects, folders, and organizations.

    • YAML_OR_JSON_SOURCE_FILE: required. The source file for the workflow in either YAML or JSON format. For example: myFirstWorkflow.yaml.

REST

To update an existing workflow, use the projects.locations.workflows.patch method and optionally use the updateMask query parameter to specify a list of fields to be updated.

Before using any of the request data, make the following replacements:

Request JSON body:

{
  "name": "WORKFLOW_NAME",
  "description": "DESCRIPTION",
  "labels": {"LABEL_KEY":"LABEL_VALUE"},
  "serviceAccount": "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com",
  "cryptoKeyName": "ENCRYPT_KEY",
  "callLogLevel": "CALL_LOGGING_LEVEL",
  "userEnvVars": {"ENV_KEY":"ENV_VALUE"},
  "sourceContents": "SOURCE_CODE"
}

To send your request, expand one of these options:

If successful, the response body contains a newly created instance of Operation:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.workflows.v1.OperationMetadata",
    "createTime": "2023-12-05T14:06:06.338390918Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/workflows/WORKFLOW_NAME",
    "verb": "update",
    "apiVersion": "v1"
  },
  "done": false
}

If the value of "done" is false, the operation is still in progress.

Delete a workflow

You can delete an existing workflow. Deleting a workflow also deletes its executions and cancels any active executions for the workflow.

Console

  1. In the Google Cloud console, go to the Workflows page.

    Go to Workflows

  2. Click the name of the workflow you want to delete and then click Delete.

  3. At the prompt to confirm the deletion, enter the workflow name.

  4. Click Confirm.

gcloud

  1. Open a terminal.

  2. Find the name of the workflow you want to delete. If you don't know the workflow's name, you can enter the following command to list all your workflows:

    gcloud workflows list
    
  3. Delete a workflow using the gcloud workflows delete command:

    gcloud workflows delete WORKFLOW_NAME
    

    Replace WORKFLOW_NAME with the name of your workflow.

REST

To delete a workflow with a specified name, use the projects.locations.workflows.delete method.

Before using any of the request data, make the following replacements:

  • WORKFLOW_NAME: the name of the workflow you want to delete.
  • PROJECT_ID: your Google Cloud project ID.
  • LOCATION: the region in which the workflow is deployed—for example, us-central1.

To send your request, expand one of these options:

If successful, the response body contains an instance of Operation:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.workflows.v1.OperationMetadata",
    "createTime": "2023-12-05T14:06:06.338390918Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/workflows/WORKFLOW_NAME",
    "verb": "delete",
    "apiVersion": "v1"
  },
  "done": false
}

If the value of "done" is false, the operation is still in progress.

What's next