Organize resources using labels

This document explains how to use labels to organize your Batch resources.

Labels are key-value pairs that are applied to resources to group and describe them. Batch has predefined labels, which are automatically applied to resources, and custom labels, which you can define and apply when creating a job.

Labels allow you to filter the results of resource lists and Cloud Billing reports. For example, you can use labels to do the following:

• Clarify and organize your project's list of jobs.

• Distinguish a job's runnables by using labels to describe the type of container or script they specify.

• Analyze costs by filtering Cloud Billing reports for the resources created by Batch or specific jobs.

For more information about labels, also see the Compute Engine documentation for labels

Before you begin

1. If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.

2. To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:

   • Batch Job Editor (roles/batch.jobsEditor) on the project
   • Service Account User (roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account

   For more information about granting roles, see Manage access to projects, folders, and organizations.

   You might also be able to get the required permissions through custom roles or other predefined roles.

Restrictions

In addition to the requirements for labels specified in the Compute Engine documentation, applying labels to a Batch job and its resources has the following restrictions:

• Batch only supports labels for resources that are created using Batch and of the following types:

  • Jobs

  • Runnables

  • Graphics processing units (GPUs) for a job (if any)

  • Persistent disks (boot disks and any storage volumes) for a job

  • Virtual machine (VM) instances for a job

• After accounting for the predefined labels that Batch automatically applies to a job, you can define the following amounts of custom labels:

  • You can define a maximum of 63 custom labels to apply to the job and its runnables.

  • You can define a maximum of 61 custom labels to apply to each GPU, persistent disk, and VM created for the job.

• Batch only supports defining custom labels with unique names. This has the following consequences:

  • Attempting to override a predefined label causes errors.

  • Defining a duplicate custom label overrides the existing custom label.

• Batch only supports defining labels when creating a job.

  • Labels for jobs and runnables can't be added, updated, or removed.

  • Although it's possible to use Compute Engine to add, update, or remove labels for the persistent disks and VMs created for jobs, this is not recommended. The timeframe that the resources for a job exist can't be reliably estimated, and any changes might not work correctly with Batch.

• To use labels to filter your list of jobs, you must view your list of jobs using the gcloud CLI or Batch API.

Predefined labels

Each predefined label has a key that begins with the batch- prefix. By default, Batch automatically applies the following predefined labels:

• To each job you create:

  • batch-job-id: The value of this label is set to the job's name.

• To each GPU, persistent disk, and VM created for a job:

  • batch-job-id: The value of this label is set to the job's name.

  • batch-job-uid: The value of this label is set to the job's unique identifier (UID).

  • batch-node: The value of this label is null—it just groups all of the GPUs, persistent disks, and VMs that are created for jobs. For example, use this label when you view a Cloud Billing report to identify the costs of all the GPUs, persistent disks, and VMs created by Batch.

Define custom labels

You can optionally define one or more custom labels when creating a job. You can define custom labels with new keys or keys that your project already uses. To define custom labels, select one or more of the following methods in this document based on the label's purpose:

• Define custom labels for the job and its resources.

  This section explains how to apply one or more custom labels to the job and to each GPU, persistent disk, and VM created for the job. After creating the job, you can use these labels to filter Cloud Billing reports and your project's lists of jobs, persistent disks, and VMs.

  Note: The persistent disks and VMs created for a job only appear in your project's lists during the job's run time (after they are created and before they are deleted).

• Define custom labels for the job.

  This section explains how to apply one or more custom labels to the job. After creating the job, you can use these labels to filter your project's lists of jobs.

• Define custom labels for runnables.

  This section explains how to apply one or more custom labels to one or more runnables for the job. After creating the job, you can use these labels to filter your project's lists of jobs.

Define custom labels for the job and its resources

Labels defined in the labels field for a job's allocation policy are applied to the job, as well as to each GPU (if any), persistent disk (all boot disks and any new storage volumes), and VM created for the job.

You can define labels for a job and its resources when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job and the resources created for the job, follow these steps:

1. Create a JSON file that specifies the job's configuration details and the allocationPolicy.labels field.

   {
     "allocationPolicy": {
       "instances": [
         {
           "policy": {
             "machineType": "e2-standard-4"
           }
         }
       ],
       "labels": {
         "VM_LABEL_NAME1": "VM_LABEL_VALUE1",
         "VM_LABEL_NAME2": "VM_LABEL_VALUE2"
       }
     },
     "taskGroups": [
       {
         "taskSpec": {
           "runnables": [
             {
               "container": {
                 "imageUri": "gcr.io/google-containers/busybox",
                 "entrypoint": "/bin/sh",
                 "commands": [
                   "-c",
                   "echo Hello world!"
                 ]
               }
             }
           ]
         }
       }
     ]
   }
   

   Replace the following:

   • VM_LABEL_NAME1: The name of the first label to apply to the VMs created for the job.

   • VM_LABEL_VALUE1: The value of the first label to apply to the VMs created for the job.

   • VM_LABEL_NAME2: The name of the second label to apply to the VMs created for the job.

   • VM_LABEL_VALUE2: The value of the second label to apply to the VMs created for the job.

   Note: To avoid confusing custom labels with predefined labels applied by Batch or other Google Cloud services, it's recommended to not specify label names that start with batch-, goog-, or google-.

2. Create the job in us-central1 using the gcloud batch jobs submit command.

   gcloud batch jobs submit example-job \
       --config=JSON_CONFIGURATION_FILE \
       --location=us-central1
   

   Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job and the resources created for the job, make a POST request to the jobs.create method and specify the allocationPolicy.labels field.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "allocationPolicy": {
    "instances": [
      {
        "policy": {
          "machineType": "e2-standard-4"
        }
      }
    ],
    "labels": {
      "VM_LABEL_NAME1": "VM_LABEL_VALUE1",
      "VM_LABEL_NAME2": "VM_LABEL_VALUE2"
    }
  },
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello world!"
              ]
            }
          }
        ]
      }
    }
  ]
}

Replace the following:

• VM_LABEL_NAME1: The name of the first label to apply to the VMs created for the job.

• VM_LABEL_VALUE1: The value of the first label to apply to the VMs created for the job.

• VM_LABEL_NAME2: The name of the second label to apply to the VMs created for the job.

• VM_LABEL_VALUE2: The value of the second label to apply to the VMs created for the job.

Note: To avoid confusing custom labels with predefined labels applied by Batch or other Google Cloud services, it's recommended to not specify label names that start with batch-, goog-, or google-.

Java

import com.google.cloud.batch.v1.AllocationPolicy;
import com.google.cloud.batch.v1.BatchServiceClient;
import com.google.cloud.batch.v1.ComputeResource;
import com.google.cloud.batch.v1.CreateJobRequest;
import com.google.cloud.batch.v1.Job;
import com.google.cloud.batch.v1.LogsPolicy;
import com.google.cloud.batch.v1.Runnable;
import com.google.cloud.batch.v1.TaskGroup;
import com.google.cloud.batch.v1.TaskSpec;
import com.google.protobuf.Duration;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class CreateBatchAllocationPolicyLabel {

  public static void main(String[] args)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // TODO(developer): Replace these variables before running the sample.
    // Project ID or project number of the Google Cloud project you want to use.
    String projectId = "YOUR_PROJECT_ID";
    // Name of the region you want to use to run the job. Regions that are
    // available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
    String region = "us-central1";
    // The name of the job that will be created.
    // It needs to be unique for each project and region pair.
    String jobName = "example-job";
    // Name of the label1 to be applied for your Job.
    String labelName1 = "VM_LABEL_NAME1";
    // Value for the label1 to be applied for your Job.
    String labelValue1 = "VM_LABEL_VALUE1";
    // Name of the label2 to be applied for your Job.
    String labelName2 = "VM_LABEL_NAME2";
    // Value for the label2 to be applied for your Job.
    String labelValue2 = "VM_LABEL_VALUE2";

    createBatchAllocationPolicyLabel(projectId, region, jobName, labelName1,
        labelValue1, labelName2, labelValue2);
  }

  // This method shows how to create a job with labels defined 
  // in the labels field of a job's allocation policy. These are 
  // applied to the job, as well as to each GPU (if any), persistent disk 
  // (all boot disks and any new storage volumes), and VM created for the job.
  public static Job createBatchAllocationPolicyLabel(String projectId, String region,
                               String jobName, String labelName1,
                               String labelValue1, String labelName2, String labelValue2)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests.
    try (BatchServiceClient batchServiceClient = BatchServiceClient.create()) {

      // Define what will be done as part of the job.
      Runnable runnable =
          Runnable.newBuilder()
              .setContainer(
                  Runnable.Container.newBuilder()
                      .setImageUri("gcr.io/google-containers/busybox")
                      .setEntrypoint("/bin/sh")
                      .addCommands("-c")
                      .addCommands(
                          "echo Hello world! This is task ${BATCH_TASK_INDEX}. "
                              + "This job has a total of ${BATCH_TASK_COUNT} tasks.")
                      .build())
              .build();

      // We can specify what resources are requested by each task.
      ComputeResource computeResource =
          ComputeResource.newBuilder()
              // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
              .setCpuMilli(2000)
              // In MiB.
              .setMemoryMib(2000)
              .build();

      TaskSpec task =
          TaskSpec.newBuilder()
              // Jobs can be divided into tasks. In this case, we have only one task.
              .addRunnables(runnable)
              .setComputeResource(computeResource)
              .setMaxRetryCount(2)
              .setMaxRunDuration(Duration.newBuilder().setSeconds(3600).build())
              .build();

      // Tasks are grouped inside a job using TaskGroups.
      // Currently, it's possible to have only one task group.
      TaskGroup taskGroup = TaskGroup.newBuilder().setTaskCount(1).setTaskSpec(task).build();

      // Policies are used to define on what kind of virtual machines the tasks will run on.
      // In this case, we tell the system to use "e2-standard-4" machine type.
      // Read more about machine types here: https://cloud.google.com/compute/docs/machine-types
      AllocationPolicy.InstancePolicy instancePolicy =
          AllocationPolicy.InstancePolicy.newBuilder().setMachineType("e2-standard-4").build();

      AllocationPolicy allocationPolicy =
          AllocationPolicy.newBuilder()
              .addInstances(AllocationPolicy.InstancePolicyOrTemplate.newBuilder()
                  .setPolicy(instancePolicy)
                  .build())
              // Labels and their value to be applied to the job and its resources
              .putLabels(labelName1, labelValue1)
              .putLabels(labelName2, labelValue2)
              .build();

      Job job =
          Job.newBuilder()
              .addTaskGroups(taskGroup)
              .setAllocationPolicy(allocationPolicy)
              // We use Cloud Logging as it's an out of the box available option.
              .setLogsPolicy(LogsPolicy.newBuilder()
                      .setDestination(LogsPolicy.Destination.CLOUD_LOGGING).build())
              .build();

      CreateJobRequest createJobRequest =
          CreateJobRequest.newBuilder()
              // The job's parent is the region in which the job will run.
              .setParent(String.format("projects/%s/locations/%s", projectId, region))
              .setJob(job)
              .setJobId(jobName)
              .build();

      Job result =
          batchServiceClient
              .createJobCallable()
              .futureCall(createJobRequest)
              .get(5, TimeUnit.MINUTES);

      System.out.printf("Successfully created the job: %s", result.getName());

      return result;
    }
  }

}

Node.js

// Imports the Batch library
const batchLib = require('@google-cloud/batch');
const batch = batchLib.protos.google.cloud.batch.v1;

// Instantiates a client
const batchClient = new batchLib.v1.BatchServiceClient();

/**
 * TODO(developer): Update these variables before running the sample.
 */
// Project ID or project number of the Google Cloud project you want to use.
const projectId = await batchClient.getProjectId();
// Name of the region you want to use to run the job. Regions that are
// available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
const region = 'europe-central2';
// The name of the job that will be created.
// It needs to be unique for each project and region pair.
const jobName = 'batch-labels-allocation-job';
// Name of the label1 to be applied for your Job.
const labelName1 = 'vm_label_name_1';
// Value for the label1 to be applied for your Job.
const labelValue1 = 'vmLabelValue1';
// Name of the label2 to be applied for your Job.
const labelName2 = 'vm_label_name_2';
// Value for the label2 to be applied for your Job.
const labelValue2 = 'vmLabelValue2';

// Define what will be done as part of the job.
const runnable = new batch.Runnable({
  script: new batch.Runnable.Script({
    commands: ['-c', 'echo Hello world! This is task ${BATCH_TASK_INDEX}.'],
  }),
});

// Specify what resources are requested by each task.
const computeResource = new batch.ComputeResource({
  // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
  cpuMilli: 500,
  // In MiB.
  memoryMib: 16,
});

const task = new batch.TaskSpec({
  runnables: [runnable],
  computeResource,
  maxRetryCount: 2,
  maxRunDuration: {seconds: 3600},
});

// Tasks are grouped inside a job using TaskGroups.
const group = new batch.TaskGroup({
  taskCount: 3,
  taskSpec: task,
});

// Policies are used to define on what kind of virtual machines the tasks will run on.
// In this case, we tell the system to use "e2-standard-4" machine type.
// Read more about machine types here: https://cloud.google.com/compute/docs/machine-types
const instancePolicy = new batch.AllocationPolicy.InstancePolicy({
  machineType: 'e2-standard-4',
});

const allocationPolicy = new batch.AllocationPolicy({
  instances: [{policy: instancePolicy}],
});
// Labels and their value to be applied to the job and its resources.
allocationPolicy.labels[labelName1] = labelValue1;
allocationPolicy.labels[labelName2] = labelValue2;

const job = new batch.Job({
  name: jobName,
  taskGroups: [group],
  labels: {env: 'testing', type: 'script'},
  allocationPolicy,
  // We use Cloud Logging as it's an option available out of the box
  logsPolicy: new batch.LogsPolicy({
    destination: batch.LogsPolicy.Destination.CLOUD_LOGGING,
  }),
});

// The job's parent is the project and region in which the job will run
const parent = `projects/${projectId}/locations/${region}`;

async function callCreateBatchLabelsAllocation() {
  // Construct request
  const request = {
    parent,
    jobId: jobName,
    job,
  };

  // Run request
  const [response] = await batchClient.createJob(request);
  console.log(JSON.stringify(response));
}

await callCreateBatchLabelsAllocation();

Python

from google.cloud import batch_v1


def create_job_with_custom_allocation_policy_labels(
    project_id: str, region: str, job_name: str, labels: dict
) -> batch_v1.Job:
    """
    This method shows the creation of a Batch job with custom labels which describe the allocation policy.
    Args:
        project_id (str): project ID or project number of the Cloud project you want to use.
        region (str): name of the region you want to use to run the job. Regions that are
            available for Batch are listed on: https://cloud.google.com/batch/docs/locations
        job_name (str): the name of the job that will be created.
        labels (dict): a dictionary of key-value pairs that will be used as labels
            E.g., {"label_key1": "label_value2", "label_key2": "label_value2"}
    Returns:
        batch_v1.Job: The created Batch job object containing configuration details.
    """
    client = batch_v1.BatchServiceClient()

    runnable = batch_v1.Runnable()
    runnable.container = batch_v1.Runnable.Container()
    runnable.container.image_uri = "gcr.io/google-containers/busybox"
    runnable.container.entrypoint = "/bin/sh"
    runnable.container.commands = [
        "-c",
        "echo Hello world!",
    ]

    # Create a task specification and assign the runnable and volume to it
    task = batch_v1.TaskSpec()
    task.runnables = [runnable]

    # Specify what resources are requested by each task.
    resources = batch_v1.ComputeResource()
    resources.cpu_milli = 2000  # in milliseconds per cpu-second. This means the task requires 2 whole CPUs.
    resources.memory_mib = 16  # in MiB
    task.compute_resource = resources

    task.max_retry_count = 2
    task.max_run_duration = "3600s"

    # Create a task group and assign the task specification to it
    group = batch_v1.TaskGroup()
    group.task_count = 3
    group.task_spec = task

    # Policies are used to define on what kind of virtual machines the tasks will run on.
    # In this case, we tell the system to use "e2-standard-4" machine type.
    # Read more about machine types here: https://cloud.google.com/compute/docs/machine-types
    policy = batch_v1.AllocationPolicy.InstancePolicy()
    policy.machine_type = "e2-standard-4"
    instances = batch_v1.AllocationPolicy.InstancePolicyOrTemplate()
    instances.policy = policy
    allocation_policy = batch_v1.AllocationPolicy()
    allocation_policy.instances = [instances]

    # Assign the provided labels to the allocation policy
    allocation_policy.labels = labels

    # Create the job and assign the task group and allocation policy to it
    job = batch_v1.Job()
    job.task_groups = [group]
    job.allocation_policy = allocation_policy

    # We use Cloud Logging as it's an out of the box available option
    job.logs_policy = batch_v1.LogsPolicy()
    job.logs_policy.destination = batch_v1.LogsPolicy.Destination.CLOUD_LOGGING

    # Create the job request and set the job and job ID
    create_request = batch_v1.CreateJobRequest()
    create_request.job = job
    create_request.job_id = job_name
    # The job's parent is the region in which the job will run
    create_request.parent = f"projects/{project_id}/locations/{region}"

    return client.create_job(create_request)

Define custom labels for the job

Labels defined in the labels field for the job are only applied to the job.

You can define labels for a job when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job itself, follow these steps:

1. Create a JSON file that specifies the job's configuration details and the labels field.

   {
     "taskGroups": [
       {
         "taskSpec": {
           "runnables": [
             {
               "container": {
                 "imageUri": "gcr.io/google-containers/busybox",
                 "entrypoint": "/bin/sh",
                 "commands": [
                   "-c",
                   "echo Hello World!"
                 ]
               }
             }
           ]
         }
       }
     ],
     "labels": {
       "JOB_LABEL_NAME1": "JOB_LABEL_VALUE1",
       "JOB_LABEL_NAME2": "JOB_LABEL_VALUE2"
     }
   }
   

   Replace the following:

   • JOB_LABEL_NAME1: The name of the first label to apply to your job.

   • JOB_LABEL_VALUE1: The value of the first label to apply to your job.

   • JOB_LABEL_NAME2: The name of the second label to apply to your job.

   • JOB_LABEL_VALUE2: The value of the second label to apply to your job.

2. Create the job in us-central1 using the gcloud batch jobs submit command with the following flags:

   gcloud batch jobs submit example-job \
       --config=JSON_CONFIGURATION_FILE \
       --location=us-central1
   

   Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a container job in us-central1 that defines two custom labels to apply to the jobs itself, make a POST request to the jobs.create method and specify the labels field.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello World!"
              ]
            }
          }
        ]
      }
    }
  ],
  "labels": {
    "JOB_LABEL_NAME1": "JOB_LABEL_VALUE1",
    "JOB_LABEL_NAME2": "JOB_LABEL_VALUE2"
  }
}

Replace the following:

• JOB_LABEL_NAME1: The name of the first label to apply to your job.

• JOB_LABEL_VALUE1: The value of the first label to apply to your job.

• JOB_LABEL_NAME2: The name of the second label to apply to your job.

• JOB_LABEL_VALUE2: The value of the second label to apply to your job.

Java

import com.google.cloud.batch.v1.BatchServiceClient;
import com.google.cloud.batch.v1.ComputeResource;
import com.google.cloud.batch.v1.CreateJobRequest;
import com.google.cloud.batch.v1.Job;
import com.google.cloud.batch.v1.LogsPolicy;
import com.google.cloud.batch.v1.Runnable;
import com.google.cloud.batch.v1.TaskGroup;
import com.google.cloud.batch.v1.TaskSpec;
import com.google.protobuf.Duration;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;


public class CreateBatchLabelJob {

  public static void main(String[] args)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // TODO(developer): Replace these variables before running the sample.
    // Project ID or project number of the Google Cloud project you want to use.
    String projectId = "YOUR_PROJECT_ID";
    // Name of the region you want to use to run the job. Regions that are
    // available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
    String region = "us-central1";
    // The name of the job that will be created.
    // It needs to be unique for each project and region pair.
    String jobName = "example-job";
    // Name of the label1 to be applied for your Job.
    String labelName1 = "JOB_LABEL_NAME1";
    // Value for the label1 to be applied for your Job.
    String labelValue1 = "JOB_LABEL_VALUE1";
    // Name of the label2 to be applied for your Job.
    String labelName2 = "JOB_LABEL_NAME2";
    // Value for the label2 to be applied for your Job.
    String labelValue2 = "JOB_LABEL_VALUE2";

    createBatchLabelJob(projectId, region, jobName, labelName1,
        labelValue1, labelName2, labelValue2);
  }

  // Creates a job with labels defined in the labels field.
  public static Job createBatchLabelJob(String projectId, String region, String jobName,
                    String labelName1, String labelValue1, String labelName2, String labelValue2)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests.
    try (BatchServiceClient batchServiceClient = BatchServiceClient.create()) {

      // Define what will be done as part of the job.
      Runnable runnable =
          Runnable.newBuilder()
              .setContainer(
                  Runnable.Container.newBuilder()
                      .setImageUri("gcr.io/google-containers/busybox")
                      .setEntrypoint("/bin/sh")
                      .addCommands("-c")
                      .addCommands(
                          "echo Hello world! This is task ${BATCH_TASK_INDEX}. "
                              + "This job has a total of ${BATCH_TASK_COUNT} tasks.")
                      .build())
              .build();

      // We can specify what resources are requested by each task.
      ComputeResource computeResource =
          ComputeResource.newBuilder()
              // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
              .setCpuMilli(2000)
              // In MiB.
              .setMemoryMib(2000)
              .build();

      TaskSpec task =
          TaskSpec.newBuilder()
              // Jobs can be divided into tasks. In this case, we have only one task.
              .addRunnables(runnable)
              .setComputeResource(computeResource)
              .setMaxRetryCount(2)
              .setMaxRunDuration(Duration.newBuilder().setSeconds(3600).build())
              .build();

      // Tasks are grouped inside a job using TaskGroups.
      // Currently, it's possible to have only one task group.
      TaskGroup taskGroup = TaskGroup.newBuilder().setTaskCount(1).setTaskSpec(task).build();

      Job job =
          Job.newBuilder()
              .addTaskGroups(taskGroup)
              // We use Cloud Logging as it's an out of the box available option.
              .setLogsPolicy(LogsPolicy.newBuilder()
              .setDestination(LogsPolicy.Destination.CLOUD_LOGGING).build())
              // Labels and their value to be applied to the job.
              .putLabels(labelName1, labelValue1)
              .putLabels(labelName2, labelValue2)
              .build();

      CreateJobRequest createJobRequest =
          CreateJobRequest.newBuilder()
              // The job's parent is the region in which the job will run.
              .setParent(String.format("projects/%s/locations/%s", projectId, region))
              .setJob(job)
              .setJobId(jobName)
              .build();

      Job result =
          batchServiceClient
              .createJobCallable()
              .futureCall(createJobRequest)
              .get(5, TimeUnit.MINUTES);

      System.out.printf("Successfully created the job: %s", result.getName());

      return result;
    }
  }

}

Node.js

// Imports the Batch library
const batchLib = require('@google-cloud/batch');
const batch = batchLib.protos.google.cloud.batch.v1;

// Instantiates a client
const batchClient = new batchLib.v1.BatchServiceClient();

/**
 * TODO(developer): Update these variables before running the sample.
 */
// Project ID or project number of the Google Cloud project you want to use.
const projectId = await batchClient.getProjectId();
// Name of the region you want to use to run the job. Regions that are
// available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
const region = 'europe-central2';
// The name of the job that will be created.
// It needs to be unique for each project and region pair.
const jobName = 'batch-labels-job';
// Name of the label1 to be applied for your Job.
const labelName1 = 'job_label_name_1';
// Value for the label1 to be applied for your Job.
const labelValue1 = 'job_label_value1';
// Name of the label2 to be applied for your Job.
const labelName2 = 'job_label_name_2';
// Value for the label2 to be applied for your Job.
const labelValue2 = 'job_label_value2';

// Define what will be done as part of the job.
const runnable = new batch.Runnable({
  container: new batch.Runnable.Container({
    imageUri: 'gcr.io/google-containers/busybox',
    entrypoint: '/bin/sh',
    commands: ['-c', 'echo Hello world! This is task ${BATCH_TASK_INDEX}.'],
  }),
});

// Specify what resources are requested by each task.
const computeResource = new batch.ComputeResource({
  // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
  cpuMilli: 500,
  // In MiB.
  memoryMib: 16,
});

const task = new batch.TaskSpec({
  runnables: [runnable],
  computeResource,
  maxRetryCount: 2,
  maxRunDuration: {seconds: 3600},
});

// Tasks are grouped inside a job using TaskGroups.
const group = new batch.TaskGroup({
  taskCount: 3,
  taskSpec: task,
});

const job = new batch.Job({
  name: jobName,
  taskGroups: [group],
  // We use Cloud Logging as it's an option available out of the box
  logsPolicy: new batch.LogsPolicy({
    destination: batch.LogsPolicy.Destination.CLOUD_LOGGING,
  }),
});

// Labels and their value to be applied to the job and its resources.
job.labels[labelName1] = labelValue1;
job.labels[labelName2] = labelValue2;

// The job's parent is the project and region in which the job will run
const parent = `projects/${projectId}/locations/${region}`;

async function callCreateBatchLabelsJob() {
  // Construct request
  const request = {
    parent,
    jobId: jobName,
    job,
  };

  // Run request
  const [response] = await batchClient.createJob(request);
  console.log(JSON.stringify(response));
}

await callCreateBatchLabelsJob();

Python

from google.cloud import batch_v1


def create_job_with_custom_job_labels(
    project_id: str,
    region: str,
    job_name: str,
    labels: dict,
) -> batch_v1.Job:
    """
    This method creates a Batch job with custom labels.
    Args:
        project_id (str): project ID or project number of the Cloud project you want to use.
        region (str): name of the region you want to use to run the job. Regions that are
            available for Batch are listed on: https://cloud.google.com/batch/docs/locations
        job_name (str): the name of the job that will be created.
        labels (dict): A dictionary of custom labels to be added to the job.
            E.g., {"label_key1": "label_value2", "label_key2": "label_value2"}
    Returns:
        batch_v1.Job: The created Batch job object containing configuration details.
    """
    client = batch_v1.BatchServiceClient()

    runnable = batch_v1.Runnable()
    runnable.container = batch_v1.Runnable.Container()
    runnable.container.image_uri = "gcr.io/google-containers/busybox"
    runnable.container.entrypoint = "/bin/sh"
    runnable.container.commands = [
        "-c",
        "echo Hello world!",
    ]

    # Create a task specification and assign the runnable and volume to it
    task = batch_v1.TaskSpec()
    task.runnables = [runnable]

    # Specify what resources are requested by each task.
    resources = batch_v1.ComputeResource()
    resources.cpu_milli = 2000  # in milliseconds per cpu-second. This means the task requires 2 whole CPUs.
    resources.memory_mib = 16  # in MiB
    task.compute_resource = resources

    task.max_retry_count = 2
    task.max_run_duration = "3600s"

    # Create a task group and assign the task specification to it
    group = batch_v1.TaskGroup()
    group.task_count = 3
    group.task_spec = task

    # Policies are used to define on what kind of virtual machines the tasks will run on.
    # In this case, we tell the system to use "e2-standard-4" machine type.
    # Read more about machine types here: https://cloud.google.com/compute/docs/machine-types
    policy = batch_v1.AllocationPolicy.InstancePolicy()
    policy.machine_type = "e2-standard-4"
    instances = batch_v1.AllocationPolicy.InstancePolicyOrTemplate()
    instances.policy = policy
    allocation_policy = batch_v1.AllocationPolicy()
    allocation_policy.instances = [instances]

    # Create the job and assign the task group and allocation policy to it
    job = batch_v1.Job()
    job.task_groups = [group]
    job.allocation_policy = allocation_policy

    # Set the labels for the job
    job.labels = labels

    # We use Cloud Logging as it's an out of the box available option
    job.logs_policy = batch_v1.LogsPolicy()
    job.logs_policy.destination = batch_v1.LogsPolicy.Destination.CLOUD_LOGGING

    # Create the job request and set the job and job ID
    create_request = batch_v1.CreateJobRequest()
    create_request.job = job
    create_request.job_id = job_name
    # The job's parent is the region in which the job will run
    create_request.parent = f"projects/{project_id}/locations/{region}"

    return client.create_job(create_request)

Define custom labels for runnables

Labels defined in the labels field for a runnable are only applied to that runnable.

You can define labels for one or more runnables when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a job in us-central1 that defines two custom labels, one custom label for each of the two job's runnables, follow these steps:

1. Create a JSON file that specifies the job's configuration details and the runnables.labels fields.

   {
     "taskGroups": [
       {
         "taskSpec": {
           "runnables": [
             {
               "container": {
                 "imageUri": "gcr.io/google-containers/busybox",
                 "entrypoint": "/bin/sh",
                 "commands": [
                   "-c",
                   "echo Hello from task ${BATCH_TASK_INDEX}!"
                 ]
               },
               "labels": {
                 "RUNNABLE1_LABEL_NAME1": "RUNNABLE1_LABEL_VALUE1"
               }
             },
             {
               "script": {
                 "text": "echo Hello from task ${BATCH_TASK_INDEX}!"
               },
               "labels": {
                 "RUNNABLE2_LABEL_NAME1": "RUNNABLE2_LABEL_VALUE1"
               }
             }
           ]
         }
       }
     ]
   }
   

   Replace the following:

   • RUNNABLE1_LABEL_NAME1: The name of the label to apply to the first runnable of the job.

   • RUNNABLE1_LABEL_VALUE1: The value of the label to apply to the first runnable of the job.

   • RUNNABLE2_LABEL_NAME1: The name of the label to apply to the second runnable of the job.

   • RUNNABLE2_LABEL_VALUE1: The value of the label to apply to the second runnable of the job.

2. Create the job in us-central1 using the gcloud batch jobs submit command.

   gcloud batch jobs submit example-job \
       --config=JSON_CONFIGURATION_FILE \
       --location=us-central1
   

   Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a job in us-central1 that defines two custom labels, one for each of the two job's runnables, make a POST request to the jobs.create method and specify the runnables.labels fields.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello from ${BATCH_TASK_INDEX}!"
              ]
            },
            "labels": {
              "RUNNABLE1_LABEL_NAME1": "RUNNABLE1_LABEL_VALUE1"
            }
          },
          {
            "script": {
              "text": "echo Hello from ${BATCH_TASK_INDEX}!"
            },
            "labels": {
              "RUNNABLE2_LABEL_NAME1": "RUNNABLE2_LABEL_VALUE1"
            }
          }
        ]
      }
    }
  ]
}

Replace the following:

• RUNNABLE1_LABEL_NAME1: The name of the label to apply to the first job's runnable.

• RUNNABLE1_LABEL_VALUE1: The value of the label to apply to the first job's runnable.

• RUNNABLE2_LABEL_NAME1: The name of the label to apply to the second job's runnable.

• RUNNABLE2_LABEL_VALUE1: The value of the label to apply to the second job's runnable.

Java

import com.google.cloud.batch.v1.BatchServiceClient;
import com.google.cloud.batch.v1.ComputeResource;
import com.google.cloud.batch.v1.CreateJobRequest;
import com.google.cloud.batch.v1.Job;
import com.google.cloud.batch.v1.LogsPolicy;
import com.google.cloud.batch.v1.Runnable;
import com.google.cloud.batch.v1.TaskGroup;
import com.google.cloud.batch.v1.TaskSpec;
import com.google.protobuf.Duration;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class CreateBatchRunnableLabel {
  public static void main(String[] args)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // TODO(developer): Replace these variables before running the sample.
    // Project ID or project number of the Google Cloud project you want to use.
    String projectId = "YOUR_PROJECT_ID";
    // Name of the region you want to use to run the job. Regions that are
    // available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
    String region = "us-central1";
    // The name of the job that will be created.
    // It needs to be unique for each project and region pair.
    String jobName = "example-job";
    // Name of the label1 to be applied for your Job.
    String labelName1 = "RUNNABLE_LABEL_NAME1";
    // Value for the label1 to be applied for your Job.
    String labelValue1 = "RUNNABLE_LABEL_VALUE1";
    // Name of the label2 to be applied for your Job.
    String labelName2 = "RUNNABLE_LABEL_NAME2";
    // Value for the label2 to be applied for your Job.
    String labelValue2 = "RUNNABLE_LABEL_VALUE2";

    createBatchRunnableLabel(projectId, region, jobName, labelName1,
        labelValue1, labelName2, labelValue2);
  }

  // Creates a job with labels defined in the labels field
  // for a runnable. The labels are only applied to that runnable.
  // In Batch, a runnable represents a single task or unit of work within a job.
  // It can be a container (like a Docker image) or a script.
  public static Job createBatchRunnableLabel(String projectId, String region, String jobName,
                   String labelName1, String labelValue1, String labelName2, String labelValue2)
      throws IOException, ExecutionException, InterruptedException, TimeoutException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests.
    try (BatchServiceClient batchServiceClient = BatchServiceClient.create()) {

      // Define what will be done as part of the job.
      Runnable runnable =
          Runnable.newBuilder()
              .setContainer(
                  Runnable.Container.newBuilder()
                      .setImageUri("gcr.io/google-containers/busybox")
                      .setEntrypoint("/bin/sh")
                      .addCommands("-c")
                      .addCommands(
                          "echo Hello world! This is task ${BATCH_TASK_INDEX}. "
                              + "This job has a total of ${BATCH_TASK_COUNT} tasks.")
                      .build())
              // Label and its value to be applied to the container
              // that processes data from a specific region.
              .putLabels(labelName1, labelValue1)
              .setScript(Runnable.Script.newBuilder()
              .setText("echo Hello world! This is task ${BATCH_TASK_INDEX}. ").build())
              // Label and its value to be applied to the script
              // that performs some analysis on the processed data.
              .putLabels(labelName2, labelValue2)
              .build();

      // We can specify what resources are requested by each task.
      ComputeResource computeResource =
          ComputeResource.newBuilder()
              // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
              .setCpuMilli(2000)
              // In MiB.
              .setMemoryMib(2000)
              .build();

      TaskSpec task =
          TaskSpec.newBuilder()
              // Jobs can be divided into tasks. In this case, we have only one task.
              .addRunnables(runnable)
              .setComputeResource(computeResource)
              .setMaxRetryCount(2)
              .setMaxRunDuration(Duration.newBuilder().setSeconds(3600).build())
              .build();

      // Tasks are grouped inside a job using TaskGroups.
      // Currently, it's possible to have only one task group.
      TaskGroup taskGroup = TaskGroup.newBuilder().setTaskCount(1).setTaskSpec(task).build();

      Job job =
          Job.newBuilder()
              .addTaskGroups(taskGroup)
              // We use Cloud Logging as it's an out of the box available option.
              .setLogsPolicy(LogsPolicy.newBuilder()
              .setDestination(LogsPolicy.Destination.CLOUD_LOGGING).build())
              .build();

      CreateJobRequest createJobRequest =
          CreateJobRequest.newBuilder()
              // The job's parent is the region in which the job will run for the specific project.
              .setParent(String.format("projects/%s/locations/%s", projectId, region))
              .setJob(job)
              .setJobId(jobName)
              .build();

      Job result =
          batchServiceClient
              .createJobCallable()
              .futureCall(createJobRequest)
              .get(5, TimeUnit.MINUTES);

      System.out.printf("Successfully created the job: %s", result.getName());

      return result;
    }
  }

}

Node.js

// Imports the Batch library
const batchLib = require('@google-cloud/batch');
const batch = batchLib.protos.google.cloud.batch.v1;

// Instantiates a client
const batchClient = new batchLib.v1.BatchServiceClient();

/**
 * TODO(developer): Update these variables before running the sample.
 */
// Project ID or project number of the Google Cloud project you want to use.
const projectId = await batchClient.getProjectId();
// Name of the region you want to use to run the job. Regions that are
// available for Batch are listed on: https://cloud.google.com/batch/docs/get-started#locations
const region = 'us-central1';
// The name of the job that will be created.
// It needs to be unique for each project and region pair.
const jobName = 'example-job';
// Name of the label1 to be applied for your Job.
const labelName1 = 'RUNNABLE_LABEL_NAME1';
// Value for the label1 to be applied for your Job.
const labelValue1 = 'RUNNABLE_LABEL_VALUE1';
// Name of the label2 to be applied for your Job.
const labelName2 = 'RUNNABLE_LABEL_NAME2';
// Value for the label2 to be applied for your Job.
const labelValue2 = 'RUNNABLE_LABEL_VALUE2';

const container = new batch.Runnable.Container({
  imageUri: 'gcr.io/google-containers/busybox',
  entrypoint: '/bin/sh',
  commands: ['-c', 'echo Hello world! This is task ${BATCH_TASK_INDEX}.'],
});

const script = new batch.Runnable.Script({
  commands: ['-c', 'echo Hello world! This is task ${BATCH_TASK_INDEX}.'],
});

const runnable1 = new batch.Runnable({
  container,
  // Label and its value to be applied to the container
  // that processes data from a specific region.
  labels: {
    [labelName1]: labelValue1,
  },
});

const runnable2 = new batch.Runnable({
  script,
  // Label and its value to be applied to the script
  // that performs some analysis on the processed data.
  labels: {
    [labelName2]: labelValue2,
  },
});

// Specify what resources are requested by each task.
const computeResource = new batch.ComputeResource({
  // In milliseconds per cpu-second. This means the task requires 50% of a single CPUs.
  cpuMilli: 500,
  // In MiB.
  memoryMib: 16,
});

const task = new batch.TaskSpec({
  runnables: [runnable1, runnable2],
  computeResource,
  maxRetryCount: 2,
  maxRunDuration: {seconds: 3600},
});

// Tasks are grouped inside a job using TaskGroups.
const group = new batch.TaskGroup({
  taskCount: 3,
  taskSpec: task,
});

const job = new batch.Job({
  name: jobName,
  taskGroups: [group],
  // We use Cloud Logging as it's an option available out of the box
  logsPolicy: new batch.LogsPolicy({
    destination: batch.LogsPolicy.Destination.CLOUD_LOGGING,
  }),
});

// The job's parent is the project and region in which the job will run
const parent = `projects/${projectId}/locations/${region}`;

async function callCreateBatchLabelsRunnable() {
  // Construct request
  const request = {
    parent,
    jobId: jobName,
    job,
  };

  // Run request
  const [response] = await batchClient.createJob(request);
  console.log(JSON.stringify(response));
}

await callCreateBatchLabelsRunnable();

Python

from google.cloud import batch_v1


def create_job_with_custom_runnables_labels(
    project_id: str,
    region: str,
    job_name: str,
    labels: dict,
) -> batch_v1.Job:
    """
    This method creates a Batch job with custom labels for runnable.
    Args:
        project_id (str): project ID or project number of the Cloud project you want to use.
        region (str): name of the region you want to use to run the job. Regions that are
            available for Batch are listed on: https://cloud.google.com/batch/docs/locations
        job_name (str): the name of the job that will be created.
        labels (dict): a dictionary of key-value pairs that will be used as labels
            E.g., {"label_key1": "label_value2"}
    Returns:
        batch_v1.Job: The created Batch job object containing configuration details.
    """
    client = batch_v1.BatchServiceClient()

    runnable = batch_v1.Runnable()
    runnable.display_name = "Script 1"
    runnable.script = batch_v1.Runnable.Script()
    runnable.script.text = "echo Hello world from Script 1 for task ${BATCH_TASK_INDEX}"
    # Add custom labels to the first runnable
    runnable.labels = labels

    # Create a task specification and assign the runnable and volume to it
    task = batch_v1.TaskSpec()
    task.runnables = [runnable]

    # Specify what resources are requested by each task.
    resources = batch_v1.ComputeResource()
    resources.cpu_milli = 2000  # in milliseconds per cpu-second. This means the task requires 2 whole CPUs.
    resources.memory_mib = 16  # in MiB
    task.compute_resource = resources

    task.max_retry_count = 2
    task.max_run_duration = "3600s"

    # Create a task group and assign the task specification to it
    group = batch_v1.TaskGroup()
    group.task_count = 3
    group.task_spec = task

    # Policies are used to define on what kind of virtual machines the tasks will run on.
    # In this case, we tell the system to use "e2-standard-4" machine type.
    # Read more about machine types here: https://cloud.google.com/compute/docs/machine-types
    policy = batch_v1.AllocationPolicy.InstancePolicy()
    policy.machine_type = "e2-standard-4"
    instances = batch_v1.AllocationPolicy.InstancePolicyOrTemplate()
    instances.policy = policy
    allocation_policy = batch_v1.AllocationPolicy()
    allocation_policy.instances = [instances]

    # Create the job and assign the task group and allocation policy to it
    job = batch_v1.Job()
    job.task_groups = [group]
    job.allocation_policy = allocation_policy

    # We use Cloud Logging as it's an out of the box available option
    job.logs_policy = batch_v1.LogsPolicy()
    job.logs_policy.destination = batch_v1.LogsPolicy.Destination.CLOUD_LOGGING

    # Create the job request and set the job and job ID
    create_request = batch_v1.CreateJobRequest()
    create_request.job = job
    create_request.job_id = job_name
    # The job's parent is the region in which the job will run
    create_request.parent = f"projects/{project_id}/locations/{region}"

    return client.create_job(create_request)

What's next

• Use labels as filters when you do the following:

  • View a list of your jobs.

  • View Cloud Billing reports.

  • View a list of persistent disks or VMs.

• Learn how to delete jobs.