Protect sensitive data using Secret Manager with Batch

This document describes how to protect sensitive data that you want to specify for a Batch job by using Secret Manager secrets.

Secret Manager secrets protect sensitive data through encryption. In a Batch job, you can specify one or more existing secrets to securely pass the sensitive data they contain, which you can use to do following:

• Securely define custom environment variables that contain sensitive data.

• Securely specify the login credentials for a Docker Registry to let a job's runnables access its private container images.

Before you begin

• If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
• Create a secret or identify a secret for the sensitive data that you want to securely specify for a job.

• To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:

  • Batch Job Editor (roles/batch.jobsEditor) on the project
  • Service Account User (roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account

  For more information about granting roles, see Manage access.

  You might also be able to get the required permissions through custom roles or other predefined roles.

• To ensure that the job's service account has the necessary permissions to access secrets, ask your administrator to grant the job's service account the Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) IAM role on the secret.

Securely pass sensitive data to custom environment variables

To securely pass sensitive data from Secret Manager secrets to custom environment variables, you must define each environment variable in the secret variables (secretVariables) subfield for an environment and specify a secret for each value. Whenever you specify a secret in a job, you must format it as a path to a secret version: projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION.

You can create a job that defines secret variables by using the gcloud CLI or Batch API. The following example explains how to create a job that defines and uses a secret variable for the environment of all runnables (environment subfield of taskSpec).

gcloud

1. Create a JSON file that specifies the job's configuration details and include the secretVariables subfield for one or more environments.

   For example, to create a basic script job that uses a secret variable in the environment for all runnables, create a JSON file with the following contents:

   Warning: For learning purposes, the following example script prints the contents of the specified secret to the standard output stream. However, scripts like this are strongly discouraged for real sensitive data, since this can cause the unencrypted contents of the secret to be recorded in unauthorized places, like Cloud Logging.

   {
     "taskGroups": [
       {
         "taskSpec": {
           "runnables": [
             {
               "script": {
                 "text": "echo This is the secret: ${SECRET_VARIABLE_NAME}"
               }
             }
           ],
           "environment": {
             "secretVariables": {
               "{SECRET_VARIABLE_NAME}": "projects/PROJECT_ID/secrets/SECRET_NAME/versions/VERSION"
             }
           }
         }
       }
     ],
     "logsPolicy": {
       "destination": "CLOUD_LOGGING"
     }
   }
   

   Replace the following:

   • SECRET_VARIABLE_NAME: the name of the secret variable. By convention, environment variable names are capitalized.

     To securely access the sensitive data from the variable's Secret Manager secret, specify this variable name in this job's runnables. The secret variable is accessible to all of the runnables that are in the same environment where you define the secret variable.

   • PROJECT_ID: the project ID of your project.

   • SECRET_NAME: the name of an existing Secret Manager secret.

   • VERSION: the version of the specified secret that contains the data you want to pass to the job. This can be the version number or latest.

2. To create and run the job, use the gcloud batch jobs submit command:

   gcloud batch jobs submit JOB_NAME \
     --location LOCATION \
     --config JSON_CONFIGURATION_FILE
   

   Replace the following:

   • JOB_NAME: the name of the job.

   • LOCATION: the location of the job.

   • JSON_CONFIGURATION_FILE: the path for a JSON file with the job's configuration details.

API

Make a POST request to the jobs.create method that specifies the secretVariables subfield for one or more environments.

For example, to create a basic script job that uses a secret variable in the environment for all runnables, make the following request:

Warning: For learning purposes, the following example script prints the contents of the specified secret to the standard output stream. However, scripts like this are strongly discouraged for real sensitive data, since this can cause the unencrypted contents of the secret to be recorded in unauthorized places, like Cloud Logging.

POST https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/jobs?job_id=JOB_NAME
{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "script": {
              "text": "echo This is the secret: ${SECRET_VARIABLE_NAME}"
            }
          }
        ],
        "environment": {
          "secretVariables": {
            "{SECRET_VARIABLE_NAME}": "projects/PROJECT_ID/secrets/SECRET_NAME/versions/VERSION"
          }
        }
      }
    }
  ],
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

Replace the following:

• PROJECT_ID: the project ID of your project.

• LOCATION: the location of the job.

• JOB_NAME: the name of the job.

• SECRET_VARIABLE_NAME: the name of the secret variable. By convention, environment variable names are capitalized.

  To securely access the sensitive data from the variable's Secret Manager secret, specify this variable name in this job's runnables. The secret variable is accessible to all of the runnables that are in the same environment where you define the secret variable.

• SECRET_NAME: the name of an existing Secret Manager secret.

• VERSION: the version of the specified secret that contains the data you want to pass to the job. This can be the version number or latest.

Securely access container images requiring Docker registry credentials

To use a container image from a private Docker registry, a runnable must specify login credentials that allow it to access that Docker registry. Specifically, for any container runnable with the image URI (imageUri) field set to a image from a private Docker registry, you must specify any credentials required to access that Docker registry by using the username (username) field and password (password) field.

You can protect any sensitive credentials for a Docker registry by specifying existing secrets that contain the information instead of defining these fields directly. Whenever you specify a secret in a job, you must format it as a path to a secret version: projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION.

You can create a job that uses container images from a private Docker registry by using the gcloud CLI or Batch API. The following example explains how to create a job that uses a container image from a private Docker registry by specifying the username directly and the password as a secret.

gcloud

1. Create a JSON file that specifies the job's configuration details. For any container runnables that use images from a private Docker registry, include any credentials required to access it in the username and password fields.

   For example, to create a basic container job that specifies an image from a private Docker registry, create a JSON file with the following contents:

   {
     "taskGroups": [
       {
         "taskSpec": {
           "runnables": [
             {
               "container": {
                 "imageUri": "PRIVATE_IMAGE_URI",
                 "commands": [
                   "-c",
                   "echo This runnable uses a private image."
                 ],
                 "username": "USERNAME",
                 "password": "PASSWORD"
               }
             }
           ],
         }
       }
     ],
     "logsPolicy": {
       "destination": "CLOUD_LOGGING"
     }
   }
   

   Replace the following:

   • PRIVATE_IMAGE_URI: the image URI for a container image from a private Docker registry. If this image requires any other container settings, you must include those also.

   • USERNAME: the username for the private Docker registry, which can be specified as a secret or directly.

   • PASSWORD: the password for the private Docker registry, which can be specified as a secret (recommended) or directly.

     For example, to specify the password as a secret, set PASSWORD to the following:

     projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION
     

     Replace the following:

     • PROJECT_ID: the project ID of your project.

     • SECRET_NAME: the name of an existing Secret Manager secret.

     • VERSION: the version of the specified secret that contains the data you want to pass to the job. This can be the version number or latest.

2. To create and run the job, use the gcloud batch jobs submit command:

   gcloud batch jobs submit JOB_NAME \
     --location LOCATION \
     --config JSON_CONFIGURATION_FILE
   

   Replace the following:

   • JOB_NAME: the name of the job.

   • LOCATION: the location of the job.

   • JSON_CONFIGURATION_FILE: the path for a JSON file with the job's configuration details.

API

Make a POST request to the jobs.create method. For any container runnables that use images from a private Docker registry, include any credentials required to access it in the username and password fields.

For example, to create a basic container job that specifies an image from a private Docker registry, make the following request:

POST https://batch.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/jobs?job_id=JOB_NAME
{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "PRIVATE_IMAGE_URI",
                "commands": [
                  "-c",
                  "echo This runnable uses a private image."
                ],
                "username": "USERNAME",
                "password": "PASSWORD"
            }
          }
        ],
      }
    }
  ],
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

Replace the following:

• PROJECT_ID: the project ID of your project.

• LOCATION: the location of the job.

• JOB_NAME: the name of the job.

• PRIVATE_IMAGE_URI: the image URI for a container image from a private Docker registry. If this image requires any other container settings, you must include those also.

• USERNAME: the username for the private Docker registry, which can be specified as a secret or directly.

• PASSWORD: the password for the private Docker registry, which can be specified as a secret (recommended) or directly.

  For example, to specify the password as a secret, set PASSWORD to the following:

  projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION
  

  Replace the following:

  • PROJECT_ID: the project ID of your project.

  • SECRET_NAME: the name of an existing Secret Manager secret.

  • VERSION: the version of the specified secret that contains the data you want to pass to the job. This can be the version number or latest.

What's next

• If you have issues creating or running a job, see Troubleshooting.

• View jobs and tasks.

• Learn more about environment variables.

• Learn more about Secret Manager.

• Learn about more job creation options.