Scheduling Workflows with Cloud Scheduler

This tutorial shows you how to use Cloud Scheduler to automatically execute Workflows so that a workflow runs on a particular schedule, in this case every 5 minutes.

Objectives

  1. Create and deploy a workflow that can receive runtime arguments.
  2. Create a Cloud Scheduler job that triggers your workflow and executes it every 5 minutes, passing in JSON arguments in the correct format.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Set up your environment for Cloud Scheduler, including creating an App Engine app.
  2. Enable the Workflows API.

    Enable the API

  3. Set the default location used in this tutorial:
      gcloud config set workflows/location REGION
    Replace REGION with the supported Workflows location of your choice.
  4. Create a service account for Workflows to use; for example, sa-name.
      gcloud iam service-accounts create sa-name
  5. Grant your service account the workflows.invoker role so that the account has permission to trigger your workflow:
      gcloud projects add-iam-policy-binding PROJECT_ID \
          --member "serviceAccount:sa-name@PROJECT_ID.iam.gserviceaccount.com" \
          --role "roles/workflows.invoker"
  6. Replace PROJECT_ID with your Google Cloud project ID.
  7. Grant the logging.logWriter role to the service account.
      gcloud projects add-iam-policy-binding PROJECT_ID \
          --member "serviceAccount:sa-name@PROJECT_ID.iam.gserviceaccount.com" \
          --role "roles/logging.logWriter"

Create a workflow that receives runtime arguments

A workflow definition is made up of a series of steps described using the Workflows syntax, which can be written in either YAML or JSON format. After creating a workflow, you deploy it to make it available for execution.

Console

  1. In the Cloud Console, go to the Workflows page:

    Go to Workflows

  2. Click Create.

  3. Enter myFirstWorkflow as a name for your new workflow.

  4. Select us-central1 for the region.

  5. Select the service account you created previously.

  6. Click Next.

  7. In the workflow editor, enter the following definition for your workflow.

    YAML

    main:
      params: [args]
      steps:
        - step1:
            assign:
              - outputVar: ${"Hello, " + args.firstName + " " + args.lastName + "!"}
        - step2:
            return: ${outputVar}
    

    JSON

    {
      "main": {
        "params": [
          "args"
        ],
        "steps": [
          {
            "step1": {
              "assign": [
                {
                  "outputVar": "${\"Hello \" + args.firstName + \" \" + args.lastName}"
                }
              ]
            }
          },
          {
            "step2": {
              "return": "${outputVar}"
            }
          }
        ]
      }
    }
      

    This workflow returns a "Hello" greeting to a person whose first and last name you pass as runtime arguments.

  8. Click Deploy.

gcloud

  1. Open a terminal.
  2. Save the following workflow definition as a YAML or JSON file, such as myFirstWorkflow.yaml or myFirstWorkflow.json.

    YAML

    main:
      params: [args]
      steps:
        - step1:
            assign:
              - outputVar: ${"Hello, " + args.firstName + " " + args.lastName + "!"}
        - step2:
            return: ${outputVar}
    

    JSON

    {
      "main": {
        "params": [
          "args"
        ],
        "steps": [
          {
            "step1": {
              "assign": [
                {
                  "outputVar": "${\"Hello \" + args.firstName + \" \" + args.lastName}"
                }
              ]
            }
          },
          {
            "step2": {
              "return": "${outputVar}"
            }
          }
        ]
      }
    }
      

    This workflow returns a "Hello" greeting to a person whose first and last name you pass as runtime arguments.

  3. Deploy the workflow by entering the following command:

    gcloud workflows deploy myFirstWorkflow \
        --source=myFirstWorkflow.yaml_OR_json \
        --service-account=sa-name@PROJECT_ID.iam.gserviceaccount.com

    Replace yaml_OR_json with yaml or json depending on the format of the Workflows definition file that you created previously.

Schedule the workflow

Create a Cloud Scheduler job that triggers your workflow, using the service account you previously created.

Console

  1. In the Cloud Console, go to the Cloud Scheduler page:

    Go to Cloud Scheduler

  2. Click Create Job.

  3. Set the Name to my-workflow-job.

  4. For Frequency, enter:

    */5 * * * *
    This will execute the job every 5 minutes. The interval is defined using unix-cron format.

  5. For Timezone, select a country and timezone.

    For example, select United States and Los Angeles or Pacific Daylight Time (PDT).

  6. Click Continue.

  7. For Target type, select HTTP.

  8. For URL, enter:

    https://workflowexecutions.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/workflows/myFirstWorkflow/executions
    

  9. Leave the HTTP method at the default of POST.

  10. Add the following two HTTP headers:

    • Name: Content-Type and Value: application/octet-stream
    • Name: User-Agent and Value: Google-Cloud-Scheduler
  11. For Body, enter:

    {"argument": "{\"firstName\":\"Sherlock\", \"lastName\":\"Holmes\"}"}
    

    This passes a JSON encoding of the workflow's arguments; in this case, the first name, "Sherlock", and the last name, "Holmes". The double quotation marks inside the string are escaped using backslashes (\).

  12. For Auth header, select Add OAuth token.

  13. Enter the service account you previously created.

    sa-name@PROJECT_ID.iam.gserviceaccount.com
    
    You don't have to specify the Scope as the default of https://www.googleapis.com/auth/cloud-platform is used.

  14. Accept all the other defaults and click Create.

gcloud

Schedule a job called my-workflow-job by entering the following command:

gcloud scheduler jobs create http my-workflow-job \
    --schedule="*/5 * * * *" \
    --uri="https://workflowexecutions.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/workflows/myFirstWorkflow/executions" \
    --message-body="{\"argument\": \"{\\\"firstName\\\":\\\"Sherlock\\\", \\\"lastName\\\":\\\"Holmes\\\"}\"}" \
    --time-zone="America/Los_Angeles" \
    --oauth-service-account-email="sa-name@PROJECT_ID.iam.gserviceaccount.com"

The interval is defined using unix-cron format.

The message-body flag is used to pass a JSON encoding of the workflow's arguments; in this case, the first name, "Sherlock", and the last name, "Holmes". The double quotation marks inside the string are escaped using backslashes (\).

Your workflow now executes every 5 minutes.

For more information about the gcloud scheduler flags, see Schedule a workflow and the gcloud reference.

Run the job and verify the results

Console

  1. In the Cloud Console, go to the Cloud Scheduler page:

    Go to Cloud Scheduler

    If the job is not displayed, you might have to refresh the page.

  2. For the job named my-workflow-job, click Run now.

    You should see the Result value update to Success.

  3. In the Cloud Console, go to the Workflows page:

    Go to Workflows

  4. From the list of workflows, click myFirstWorkflow to open its details page.

    The execution state of your workflow should be Succeeded.

  5. Click the workflow's execution ID to open the details page and view the results of the workflow in the Output pane.

    The output should be "Hello, Sherlock Holmes!".

gcloud

  1. Run the Cloud Scheduler job to execute the workflow:

    gcloud scheduler jobs run my-workflow-job
    
  2. Retrieve the execution ID for your workflow:

    gcloud workflows executions list myFirstWorkflow
    

    The output should resemble the following with the execution ID in bold:

    projects/316710615161/locations/us-central1/workflows/myFirstWorkflow/executions/138b31e1-f3bb-4ba7-a6da-008d0cb4a320
  3. Retrieve the results of the workflow's execution:

    gcloud workflows executions describe EXECUTION_ID --workflow myFirstWorkflow

    Replace EXECUTION_ID with the execution ID returned in the previous step.

    The output should resemble the following:

    argument: '{"firstName":"Sherlock","lastName":"Holmes"}'
    endTime: '2021-09-09T15:15:00.869350639Z'
    name: projects/316710615161/locations/us-central1/workflows/myFirstWorkflow/executions/138b31e1-f3bb-4ba7-a6da-008d0cb4a320
    result: '"Hello, Sherlock Holmes!"'
    startTime: '2021-09-09T15:15:00.839175480Z'
    state: SUCCEEDED
    workflowRevisionId: 000001-4f9

Congratulations! You have successfully created a Cloud Scheduler job that automatically runs a workflow on a particular schedule.

Clean up

After you finish the tutorial, you can clean up the resources that you created so that they stop using quota and incurring charges. The following sections describe how to delete or turn off these resources.

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the Cloud Scheduler job

  1. Go to the Cloud Scheduler page in the Cloud Console.

    Go to Cloud Scheduler

  2. Select the checkbox next to your job.

  3. Click Delete at the top of the page and confirm your deletion.

Delete the Workflows workflow

  1. Go to the Workflows page in the Cloud Console.

    Go to Workflows

  2. Select your workflow.

  3. Click Delete at the top of the page; type your workflow's name, and confirm your deletion.

Disable the App Engine app

Disabling an App Engine app stops your application from running instances and serving requests while retaining application data and settings.

  1. Go to the Application settings page in the Cloud Console.

    Go to Application settings

  2. Click Disable application.

  3. Follow the prompts.

What's next