Organize resources using labels

This document explains how to use labels to organize your Batch resources.

Labels are key-value pairs that are applied to resources to group and describe them. Batch has predefined labels, which are automatically applied to resources, and custom labels, which you can define and apply when creating a job.

Labels allow you to filter the results of resource lists and Cloud Billing reports. For example, you can use labels to do the following:

Clarify and organize your project's list of jobs.
Distinguish a job's runnables by using labels to describe the type of container or script they specify.
Analyze costs by filtering Cloud Billing reports for the resources created by Batch or specific jobs.

For more information about labels, also see the Compute Engine documentation for labels

Before you begin

If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:
- Batch Job Editor (roles/batch.jobsEditor) on the project
- Service Account User (roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account
For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

Restrictions

In addition to the requirements for labels specified in the Compute Engine documentation, applying labels to a Batch job and its resources has the following restrictions:

Batch only supports labels for resources that are created using Batch and of the following types:
- Jobs
- Runnables
- Graphics processing units (GPUs) for a job (if any)
- Persistent disks (boot disks and any storage volumes) for a job
- Virtual machine (VM) instances for a job
After accounting for the predefined labels that Batch automatically applies to a job, you can define the following amounts of custom labels:
- You can define a maximum of 63 custom labels to apply to the job and its runnables.
- You can define a maximum of 61 custom labels to apply to each GPU, persistent disk, and VM created for the job.
Batch only supports defining custom labels with unique names. This has the following consequences:
- Attempting to override a predefined label causes errors.
- Defining a duplicate custom label overrides the existing custom label.
Batch only supports defining labels when creating a job.
- Labels for jobs and runnables can't be added, updated, or removed.
- Although it's possible to use Compute Engine to add, update, or remove labels for the persistent disks and VMs created for jobs, this is not recommended. The timeframe that the resources for a job exist can't be reliably estimated, and any changes might not work correctly with Batch.
To use labels to filter your list of jobs, you must view your list of jobs using the gcloud CLI or Batch API.

Predefined labels

Each predefined label has a key that begins with the batch- prefix. By default, Batch automatically applies the following predefined labels:

To each job you create:
- batch-job-id: The value of this label is set to the job's name.
To each GPU, persistent disk, and VM created for a job:
- batch-job-id: The value of this label is set to the job's name.
- batch-job-uid: The value of this label is set to the job's unique identifier (UID).
- batch-node: The value of this label is null—it just groups all of the GPUs, persistent disks, and VMs that are created for jobs. For example, use this label when you view a Cloud Billing report to identify the costs of all the GPUs, persistent disks, and VMs created by Batch.

Define custom labels

You can optionally define one or more custom labels when creating a job. You can define custom labels with new keys or keys that your project already uses. To define custom labels, select one or more of the following methods in this document based on the label's purpose:

Define custom labels for the job and its resources.

This section explains how to apply one or more custom labels to the job and to each GPU, persistent disk, and VM created for the job. After creating the job, you can use these labels to filter Cloud Billing reports and your project's lists of jobs, persistent disks, and VMs.

Note: The persistent disks and VMs created for a job only appear in your project's lists during the job's run time (after they are created and before they are deleted).
Define custom labels for the job.

This section explains how to apply one or more custom labels to the job. After creating the job, you can use these labels to filter your project's lists of jobs.
Define custom labels for runnables.

This section explains how to apply one or more custom labels to one or more runnables for the job. After creating the job, you can use these labels to filter your project's lists of jobs.

Define custom labels for the job and its resources

Labels defined in the labels field for a job's allocation policy are applied to the job, as well as to each GPU (if any), persistent disk (all boot disks and any new storage volumes), and VM created for the job.

You can define labels for a job and its resources when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job and the resources created for the job, follow these steps:

Create a JSON file that specifies the job's configuration details and the allocationPolicy.labels field.
```
{
  "allocationPolicy": {
    "instances": [
      {
        "policy": {
          "machineType": "e2-standard-4"
        }
      }
    ],
    "labels": {
      "VM_LABEL_NAME1": "VM_LABEL_VALUE1",
      "VM_LABEL_NAME2": "VM_LABEL_VALUE2"
    }
  },
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello world!"
              ]
            }
          }
        ]
      }
    }
  ]
}
```
Replace the following:
- VM_LABEL_NAME1: The name of the first label to apply to the VMs created for the job.
- VM_LABEL_VALUE1: The value of the first label to apply to the VMs created for the job.
- VM_LABEL_NAME2: The name of the second label to apply to the VMs created for the job.
- VM_LABEL_VALUE2: The value of the second label to apply to the VMs created for the job.
Note: To avoid confusing custom labels with predefined labels applied by Batch or other Google Cloud services, it's recommended to not specify label names that start with batch-, goog-, or google-.
Create the job in us-central1 using the gcloud batch jobs submit command.
```
gcloud batch jobs submit example-job \
    --config=JSON_CONFIGURATION_FILE \
    --location=us-central1
```
Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job and the resources created for the job, make a POST request to the jobs.create method and specify the allocationPolicy.labels field.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "allocationPolicy": {
    "instances": [
      {
        "policy": {
          "machineType": "e2-standard-4"
        }
      }
    ],
    "labels": {
      "VM_LABEL_NAME1": "VM_LABEL_VALUE1",
      "VM_LABEL_NAME2": "VM_LABEL_VALUE2"
    }
  },
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello world!"
              ]
            }
          }
        ]
      }
    }
  ]
}

Replace the following:

VM_LABEL_NAME1: The name of the first label to apply to the VMs created for the job.
VM_LABEL_VALUE1: The value of the first label to apply to the VMs created for the job.
VM_LABEL_NAME2: The name of the second label to apply to the VMs created for the job.
VM_LABEL_VALUE2: The value of the second label to apply to the VMs created for the job.

Define custom labels for the job

Labels defined in the labels field for the job are only applied to the job.

You can define labels for a job when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a basic container job in us-central1 that defines two custom labels that apply to the job itself, follow these steps:

Create a JSON file that specifies the job's configuration details and the labels field.

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello World!"
              ]
            }
          }
        ]
      }
    }
  ],
  "labels": {
    "JOB_LABEL_NAME1": "JOB_LABEL_VALUE1",
    "JOB_LABEL_NAME2": "JOB_LABEL_VALUE2"
  }
}

Replace the following:

JOB_LABEL_NAME1: The name of the first label to apply to your job.
JOB_LABEL_VALUE1: The value of the first label to apply to your job.
JOB_LABEL_NAME2: The name of the second label to apply to your job.
JOB_LABEL_VALUE2: The value of the second label to apply to your job.

Create the job in us-central1 using the gcloud batch jobs submit command with the following flags:
```
gcloud batch jobs submit example-job \
    --config=JSON_CONFIGURATION_FILE \
    --location=us-central1
```
Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a container job in us-central1 that defines two custom labels to apply to the jobs itself, make a POST request to the jobs.create method and specify the labels field.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello World!"
              ]
            }
          }
        ]
      }
    }
  ],
  "labels": {
    "JOB_LABEL_NAME1": "JOB_LABEL_VALUE1",
    "JOB_LABEL_NAME2": "JOB_LABEL_VALUE2"
  }
}

Replace the following:

JOB_LABEL_NAME1: The name of the first label to apply to your job.
JOB_LABEL_VALUE1: The value of the first label to apply to your job.
JOB_LABEL_NAME2: The name of the second label to apply to your job.
JOB_LABEL_VALUE2: The value of the second label to apply to your job.

Define custom labels for runnables

Labels defined in the labels field for a runnable are only applied to that runnable.

You can define labels for one or more runnables when creating a job using the gcloud CLI or Batch API.

gcloud

For example, to create a job in us-central1 that defines two custom labels, one custom label for each of the two job's runnables, follow these steps:

Create a JSON file that specifies the job's configuration details and the runnables.labels fields.

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello from task ${BATCH_TASK_INDEX}!"
              ]
            },
            "labels": {
              "RUNNABLE1_LABEL_NAME1": "RUNNABLE1_LABEL_VALUE1"
            }
          },
          {
            "script": {
              "text": "echo Hello from task ${BATCH_TASK_INDEX}!"
            },
            "labels": {
              "RUNNABLE2_LABEL_NAME1": "RUNNABLE2_LABEL_VALUE1"
            }
          }
        ]
      }
    }
  ]
}

Replace the following:

RUNNABLE1_LABEL_NAME1: The name of the label to apply to the first runnable of the job.
RUNNABLE1_LABEL_VALUE1: The value of the label to apply to the first runnable of the job.
RUNNABLE2_LABEL_NAME1: The name of the label to apply to the second runnable of the job.
RUNNABLE2_LABEL_VALUE1: The value of the label to apply to the second runnable of the job.

Create the job in us-central1 using the gcloud batch jobs submit command.
```
gcloud batch jobs submit example-job \
    --config=JSON_CONFIGURATION_FILE \
    --location=us-central1
```
Replace JSON_CONFIGURATION_FILE with the path to the JSON file with the job's configuration details you created in the previous step.

API

For example, to create a job in us-central1 that defines two custom labels, one for each of the two job's runnables, make a POST request to the jobs.create method and specify the runnables.labels fields.

POST https://batch.googleapis.com/v1/projects/example-project/locations/us-central1/jobs?job_id=example-job

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "gcr.io/google-containers/busybox",
              "entrypoint": "/bin/sh",
              "commands": [
                "-c",
                "echo Hello from ${BATCH_TASK_INDEX}!"
              ]
            },
            "labels": {
              "RUNNABLE1_LABEL_NAME1": "RUNNABLE1_LABEL_VALUE1"
            }
          },
          {
            "script": {
              "text": "echo Hello from ${BATCH_TASK_INDEX}!"
            },
            "labels": {
              "RUNNABLE2_LABEL_NAME1": "RUNNABLE2_LABEL_VALUE1"
            }
          }
        ]
      }
    }
  ]
}

Replace the following:

RUNNABLE1_LABEL_NAME1: The name of the label to apply to the first job's runnable.
RUNNABLE1_LABEL_VALUE1: The value of the label to apply to the first job's runnable.
RUNNABLE2_LABEL_NAME1: The name of the label to apply to the second job's runnable.
RUNNABLE2_LABEL_VALUE1: The value of the label to apply to the second job's runnable.

What's next

Use labels as filters when you do the following:
Learn how to delete and export jobs.