Create and run Batch jobs using Terraform and Cloud Scheduler


This tutorial explains how you can use Terraform to create and run Batch jobs by using a Cloud Scheduler cron job.

Terraform is an open-source tool that lets you provision and manage infrastructure by specifying the desired state in configuration files. These files that can be treated as code and stored in version control systems like GitHub.

Although Terraform doesn't have resources for Batch, this tutorial shows how you can use Terraform to create Batch jobs. Specifically, you can use Terraform to schedule and run a Cloud Scheduler cron job that targets the Batch API to create and run Batch jobs. Cloud Scheduler is a Google Cloud service that allows you to automatically schedule cron jobs and supports Terraform.

This tutorial is intended for Batch users who already manage infrastructure with Terraform and want to incorporate Batch jobs into Terraform.

Objectives

  • Create a Terraform directory and a configuration file that defines a Cloud Scheduler cron job that creates Batch jobs.
  • Deploy the Terraform configuration to run the cron job.
  • Verify that the cron job creates Batch jobs.
  • Update the Terraform configuration to pause the cron job so that it stops creating Batch jobs.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. Prepare your development environment, either Cloud Shell or a local shell:

    Cloud Shell

    To use an online terminal with the gcloud CLI and Terraform already set up, activate Cloud Shell.

    At the bottom of this page, a Cloud Shell session starts and displays a command-line prompt. It can take a few seconds for the session to initialize.

    Local shell

    To use a local development environment, follow these steps:

    1. Install the Google Cloud CLI.
    2. To initialize the gcloud CLI, run the following command:

      gcloud init
    3. Install Terraform.
  2. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Batch, Compute Engine, Cloud Logging, Cloud Scheduler, and Resource Manager APIs:

    gcloud services enable batch.googleapis.com compute.googleapis.com logging.googleapis.com  cloudscheduler.googleapis.com cloudresourcemanager.googleapis.com
  5. Make sure that your project has at least one service account with the permissions required for this tutorial.

    Specifically, you can use either the same service account or two separate service accounts to grant the following permissions:

    • Allow the cron job to create Batch jobs and attach the service account for the Batch jobs.
    • Allow the Batch jobs to create and access the resources required to run.

    To ensure that the service account(s) for this tutorial has the necessary permissions to use Terraform to create Batch jobs through a Cloud Scheduler cron job, ask your administrator to grant the service account(s) for this tutorial the following IAM roles:

    • Service account for the Cloud Scheduler cron job:
    • Service account for the Batch jobs:

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    Your administrator might also be able to give the service account(s) for this tutorial the required permissions through custom roles or other predefined roles.

  6. Make sure that you have the permissions required for this tutorial.

    Specifically, you need permissions to do the following:

    • Create a cron job and attach the service account for the cron job.
    • View and delete the cron job and Batch jobs.

    To get the permissions that you need to use Terraform to create Batch jobs through a Cloud Scheduler cron job, ask your administrator to grant you the following IAM roles:

Create the Terraform directory and configuration file

Create a directory for Terraform and a configuration file that defines the resources that you want create or update using Terraform. The example configuration file for this tutorial defines a Cloud Scheduler cron job named batch-job-invoker. When it is enabled, the batch-job-invoker cron job runs every 5 minutes to create a new instance of the defined Batch job.

  1. To create a directory and a new Terraform configuration (.tf) file within that directory, type the following command, and then press Enter:

    mkdir terraform && cd terraform && cat > main.tf
    

    This command creates the terraform directory, navigates you to it, and starts defining a new main.tf configuration file on the next line.

  2. Copy and paste the following Terraform configuration:

    # define variables
    variable "project_id" {
      type        = string
      description = "The project name to use."
      default = "PROJECT_ID"
    }
    
    variable "project_number" {
      type        = string
      description = "The project number to use."
      default = "PROJECT_NUMBER"
    }
    
    variable "region" {
      type        = string
      description = "The region where resources are created."
      default = "us-central1"
    }
    
    variable "cloud_scheduler_service_account_email" {
      type        = string
      description = "The service account email."
      default = "CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL"
    }
    
    variable "batch_service_account_email" {
      type        = string
      description = "The service account email."
      default = "BATCH_SERVICE_ACCOUNT_EMAIL"
    }
    
    # define a Cloud Scheduler cron job which triggers Batch jobs
    resource "google_cloud_scheduler_job" "batch-job-invoker" {
      paused           = false # this cron job is enabled
      name             = "batch-job-invoker"
      project          = var.project_id
      region           = var.region
      schedule         = "*/5 * * * *" # when enabled, run every 5 minutes
      time_zone        = "America/Los_Angeles"
      attempt_deadline = "180s"
    
      retry_config {
        max_doublings        = 5
        max_retry_duration   = "0s"
        max_backoff_duration = "3600s"
        min_backoff_duration = "5s"
      }
    
      # when this cron job runs, create and run a Batch job
      http_target {
        http_method = "POST"
        uri = "https://batch.googleapis.com/v1/projects/${var.project_number}/locations/${var.region}/jobs"
        headers = {
          "Content-Type" = "application/json"
          "User-Agent"   = "Google-Cloud-Scheduler"
        }
        # Batch job definition
        body = base64encode(<<EOT
        {
          "taskGroups":[
            {
              "taskSpec": {
                "runnables":{
                  "script": {
                    "text": "echo Hello world! This job was created using Terraform and Cloud Scheduler."
                  }
                }
              }
            }
          ],
          "allocationPolicy": {
            "serviceAccount": {
              "email": "${var.batch_service_account_email}"
            }
          },
          "labels": {
            "source": "terraform_and_cloud_scheduler_tutorial"
          },
          "logsPolicy": {
            "destination": "CLOUD_LOGGING"
          }
        }
        EOT
        )
        oauth_token {
          scope                 = "https://www.googleapis.com/auth/cloud-platform"
          service_account_email = var.cloud_scheduler_service_account_email
        }
      }
    }
    
    

    Replace the following:

    • PROJECT_ID: the project ID of your project.
    • PROJECT_NUMBER: the project number of your project.
    • CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL: the email address of the service account that you prepared for the Cloud Scheduler cron job.

      For example, to use the Compute Engine default service account, specify the following:

      PROJECT_NUMBER-compute@developer.gserviceaccount.com
      
    • BATCH_SERVICE_ACCOUNT_EMAIL: the email address of the service account that you prepared for Batch jobs.

      For example, to use the Compute Engine default service account, specify the following:

      PROJECT_NUMBER-compute@developer.gserviceaccount.com
      

    This Terraform configuration defines some input variables and a cron job that contacts the API method for creating a Batch job.

  3. To save and close the file, press Ctrl+D (or Command+D on macOS).

Deploy the Terraform configuration to create the cron job

Deploy the Terraform configuration by initializing Terraform, generating the planned changes, and applying these changes. After deploying the Terraform configuration, you can describe the resources in your project to verify that Terraform successfully created the batch-job-invoker cron job.

  1. Initialize Terraform in the directory:

    terraform init
    

    The output is similar to the following:

    ...
    Terraform has been successfully initialized!
    
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.
    
    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    
  2. Generate the Terraform execution plan based on the current state of your project and the configuration file:

    terraform plan
    

    The output is similar to the following, which shows that the plan is to create the batch-job-invoker cron job:

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create
    
    Terraform will perform the following actions:
    
      # google_cloud_scheduler_job.batch-job-invoker will be created
      + resource "google_cloud_scheduler_job" "batch-job-invoker" {
          + id        = (known after apply)
          + name      = "batch-job-invoker"
          + paused    = false
          + project   = "PROJECT_ID"
          + region    = "us-central1"
          + schedule  = "*/5 * * * *"
          + state     = (known after apply)
          + time_zone = "America/Los_Angeles"
    
          + http_target {
              + body        = "..."
              + headers     = {
                  + "Content-Type" = "application/json"
                  + "User-Agent"   = "Google-Cloud-Scheduler"
                }
              + http_method = "POST"
              + uri         = "https://batch.googleapis.com/v1/projects/PROJECT_NUMBER/locations/us-central1/jobs"
    
              + oauth_token {
                  + scope                 = "https://www.googleapis.com/auth/cloud-platform"
                  + service_account_email = "CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL"
                }
            }
    
          + retry_config {
              + max_backoff_duration = "3600s"
              + max_doublings        = 5
              + max_retry_duration   = "0s"
              + min_backoff_duration = "5s"
              + retry_count          = (known after apply)
            }
        }
    
    Plan: 1 to add, 0 to change, 0 to destroy.
    
    ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    
    Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
    
  3. To apply the plan to create the batch-job-invoker cron job, follow these steps:

    1. Enter the following command:

      terraform apply
      

      The output is similar to the previous terraform plan command except that it ends with a confirmation prompt.

    2. To confirm and apply the plan, enter yes.

      The output is similar to the following:

      google_cloud_scheduler_job.batch-job-invoker: Creating...
      google_cloud_scheduler_job.batch-job-invoker: Creation complete after 0s [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      
      Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
      
  4. To verify that the batch-job-invoker cron job exists and is enabled, describe it:

    gcloud scheduler jobs describe batch-job-invoker --location us-central1
    

    The output is similar to the following:

    attemptDeadline: 180s
    httpTarget:
      body: ...
      headers:
        Content-Type: application/json
        User-Agent: Google-Cloud-Scheduler
      httpMethod: POST
      oauthToken:
        scope: https://www.googleapis.com/auth/cloud-platform
        serviceAccountEmail: CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL
      uri: https://batch.googleapis.com/v1/projects/PROJECT_NUMBER/locations/us-central1/jobs
    lastAttemptTime: '...'
    name: projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker
    retryConfig:
      maxBackoffDuration: 3600s
      maxDoublings: 5
      maxRetryDuration: 0s
      minBackoffDuration: 5s
    schedule: '*/5 * * * *'
    scheduleTime: '...'
    state: ENABLED
    status: {}
    timeZone: America/Los_Angeles
    userUpdateTime: '...'
    

    In the output, verify that the state field is set to ENABLED.

Verify that the cron job creates a Batch job

Verify that the batch-job-invoker cron job is correctly creating Batch jobs.

  1. Either wait 5 minutes for the cron job to run automatically or trigger the cron job to run immediately:

    gcloud scheduler jobs run batch-job-invoker --location us-central1
    
  2. List the Batch jobs that have been created by the batch-job-invoker cron job:

    gcloud batch jobs list \
    --filter labels.source=\"terraform_and_cloud_scheduler_tutorial\" \
    --sort-by ~createTime
    
    • The --filter labels.source=\"terraform_and_cloud_scheduler_tutorial\" flag filters the list to only include Batch jobs that have a label with the key source and the value terraform_and_cloud_scheduler_tutorial.
    • The --sort-by ~createTime flag sorts the list from newest to oldest.

Update the Terraform configuration to pause the cron job

After you have the desired number of Batch jobs, update and deploy the Terraform configuration to pause the batch-job-invoker cron job. If you want to update other properties of the cron job or future Batch jobs, this same process applies.

  1. Update the Terraform configuration file to pause the cron job by setting the paused field to true:

    sed -i 's/paused           = false # this cron job is enabled/paused           = true # this cron job is paused/g' main.tf
    
  2. Generate the Terraform execution plan based on the current state of your project and the configuration file:

    terraform plan
    

    The output is similar to the following, which shows that the plan is to update the value of paused field from false to true:

    google_cloud_scheduler_job.batch-job-invoker: Refreshing state... [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
    
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      ~ update in-place
    
    Terraform will perform the following actions:
    
      # google_cloud_scheduler_job.batch-job-invoker will be updated in-place
      ~ resource "google_cloud_scheduler_job" "batch-job-invoker" {
            id               = "projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker"
            name             = "batch-job-invoker"
          ~ paused           = false -> true
            # (6 unchanged attributes hidden)
    
          ~ http_target {
              ~ headers     = {
                  + "User-Agent"   = "Google-Cloud-Scheduler"
                    # (1 unchanged element hidden)
                }
                # (3 unchanged attributes hidden)
    
                # (1 unchanged block hidden)
            }
    
            # (1 unchanged block hidden)
        }
    
    Plan: 0 to add, 1 to change, 0 to destroy.
    
    ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    
    Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
    
  3. To apply the plan to update the batch-job-invoker cron job, follow these steps:

    1. Enter the following command:

      terraform apply
      

      The output is similar to the previous terraform plan command except that it ends with a confirmation prompt.

    2. To confirm and apply the plan, enter yes.

      The output is similar to the following:

      google_cloud_scheduler_job.batch-job-invoker: Modifying... [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      google_cloud_scheduler_job.batch-job-invoker: Modifications complete after 1s [id=projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker]
      
      Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
      
  4. To verify that the batch-job-invoker cron job is paused, describe it:

    gcloud scheduler jobs describe batch-job-invoker --location us-central1
    

    The output is similar to the following:

    attemptDeadline: 180s
    httpTarget:
      body: ...
      headers:
        Content-Type: application/json
        User-Agent: Google-Cloud-Scheduler
      httpMethod: POST
      oauthToken:
        scope: https://www.googleapis.com/auth/cloud-platform
        serviceAccountEmail: CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL
      uri: https://batch.googleapis.com/v1/projects/PROJECT_NUMBER/locations/us-central1/jobs
    lastAttemptTime: '...'
    name: projects/PROJECT_ID/locations/us-central1/jobs/batch-job-invoker
    retryConfig:
      maxBackoffDuration: 3600s
      maxDoublings: 5
      maxRetryDuration: 0s
      minBackoffDuration: 5s
    schedule: '*/5 * * * *'
    scheduleTime: '...'
    state: PAUSED
    status: {}
    timeZone: America/Los_Angeles
    userUpdateTime: '...'
    

    In the output, verify that the state field is set to PAUSED.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. Delete a Google Cloud project:

    gcloud projects delete PROJECT_ID

  2. Go to the parent directory, and then delete the Terraform directory and all of its files.

    cd .. && rm -r terraform
    

Delete individual resources

  1. Delete the batch-job-invoker cron job.

    terraform destroy
    
  2. To delete all the Batch jobs from this tutorial, follow these steps:

    1. List all the Batch jobs that were created by the batch-job-invoker cron job:

      gcloud batch jobs list \
      --filter labels.source=\"terraform_and_cloud_scheduler_tutorial\" \
      --sort-by ~createTime
      

      Record the name of each job that you need to delete.

    2. Delete a Batch job from this tutorial:

      gcloud batch jobs delete JOB_NAME --location us-central1
      

      Replace JOB_NAME with the name of a Batch job.

      Repeat this step for all the Batch jobs.

  3. If you created a service account for this tutorial, delete the service account:

    gcloud iam service-accounts delete SERVICE_ACCOUNT_EMAIL
    

    Replace SERVICE_ACCOUNT_EMAIL with the email address of a service account that you created for this tutorial. Namely, you used the following service accounts:

    • CLOUD_SCHEDULER_SERVICE_ACCOUNT_EMAIL: the service account for Cloud Scheduler.
    • BATCH_SERVICE_ACCOUNT_EMAIL: the service account for Batch.

    If you created two separate service accounts, repeat this step.

  4. Go to the parent directory, and then delete the Terraform directory and all of its files.

    cd .. && rm -r terraform
    

What's next