This document explains how to configure custom status events, which describe a job's runnables, when you create and run a Batch job. To learn about status events, see View a job's history through status events.
Custom status events let you provide additional details in a task's history about the progress of its runnables, which can help make a job easier to analyze and troubleshoot. For example, you can configure custom status events that describe when a runnable starts, a runnable ends, a barrier runnable is reached, or an important event happens during the progression of your code.
Before you begin
- If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
-
To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:
-
Batch Job Editor (
roles/batch.jobsEditor) on the project -
Service Account User (
roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Batch Job Editor (
Configure custom status events
Configure custom status events by using one or more of following options when you are creating a job:
Describe a runnable's state by defining its display name. You can do this when you create a job using the gcloud CLI or Batch API.
Indicate important runtime events by writing a structured task log with the
batch/custom/eventfield for each event. You can do this when using any method to create a job as part of the definitions of script and container runnables.
Describe a runnable's state
You can configure custom status events that describe a runnable's state by
defining a runnable's display name
(displayName field).
The resulting custom status events vary slightly for different types of
runnables:
If you define a display name for a container runnable or script runnable, then Batch automatically adds two types of custom status events. The first custom status event indicates whenever a task starts this runnable. The second custom status event indicates whenever a tasks finishes this runnable and the corresponding exit code.
If you define a display name for a barrier runnable, then Batch automatically adds a custom status event that indicates whenever a task reaches this barrier.
To create and run a job with custom status events that describes a
runnable's state, define the displayName field for one or more
runnables using the gcloud CLI, Batch API or
library.
gcloud
Use the Google Cloud CLI to
create a job that
includes the displayName field in one or more runnables definitions
in the JSON file:
...
"runnables": [
{
"displayName":DISPLAY_NAME,
...
}
]
...
For example, a job with custom status events that describes each runnable's state can have a JSON configuration file similar to the following:
{
"taskGroups": [
{
"taskSpec": {
"runnables": [
{
"displayName":"DISPLAY_NAME1",
"script": {
"text": "echo Hello world from script 1 for task ${BATCH_TASK_INDEX}"
}
},
{
"displayName":"DISPLAY_NAME2",
"barrier": {}
},
{
"displayName":"DISPLAY_NAME3",
"script": {
"text": "echo Hello world from script 2 for task ${BATCH_TASK_INDEX}"
}
}
]
},
"taskCount": 3
}
],
"logsPolicy": {
"destination": "CLOUD_LOGGING"
}
}
Replace DISPLAY_NAME1,
DISPLAY_NAME2, and
DISPLAY_NAME3 with the name of the runnable, which
must be unique within the job—for example, script 1, barrier 1, and
script 2.
API
Use the REST API to
create a job that
includes the displayName field in one or more runnables definitions
in the JSON file:
...
"runnables": [
{
"displayName":DISPLAY_NAME,
...
}
]
...
For example, a job with custom status events that describes each runnable's state can have a JSON configuration file similar to the following:
{
"taskGroups": [
{
"taskSpec": {
"runnables": [
{
"displayName":"DISPLAY_NAME1",
"script": {
"text": "echo Hello world from script 1 for task ${BATCH_TASK_INDEX}"
}
},
{
"displayName":"DISPLAY_NAME2",
"barrier": {}
},
{
"displayName":"DISPLAY_NAME3",
"script": {
"text": "echo Hello world from script 2 for task ${BATCH_TASK_INDEX}"
}
}
]
},
"taskCount": 3
}
],
"logsPolicy": {
"destination": "CLOUD_LOGGING"
}
}
Replace DISPLAY_NAME1,
DISPLAY_NAME2, and
DISPLAY_NAME3 with the name of the runnable, which
must be unique within the job—for example, script 1, barrier 1, and
script 2.
Go
Java
Node.js
Python
After the example job has finished running, the resulting custom status events for each task are similar to the following:
statusEvents:
...
- description: 'script at index #0 with display name [DISPLAY_NAME1] started.'
eventTime: '...'
type: RUNNABLE_EVENT
- description: 'script at index #0 with display name [DISPLAY_NAME1] finished with exit
code 0.'
eventTime: '...'
type: RUNNABLE_EVENT
- description: 'barrier at index #2 with display name [DISPLAY_NAME2] reached.'
eventTime: '...'
type: BARRIER_REACHED_EVENT
- description: 'script at index #2 with display name [DISPLAY_NAME3] started.'
eventTime: '...'
type: RUNNABLE_EVENT
- description: 'script at index #2 with display name [DISPLAY_NAME3] finished with exit
code 0.'
eventTime: '...'
type: RUNNABLE_EVENT
...
Indicate important runtime events
You can configure custom status events that indicate when an important event
happens while a runnable is running by configuring that runnable to write a
structured task log that defines a string for the Batch
custom status event (batch/custom/event) field.
If a container runnable or script runnable writes a structured task log
that defines the batch/custom/event JSON field, it produces a
custom status event at that time. Although you might configure the structured
task log to include additional fields, the custom status event only includes
the string for the batch/custom/event field.
To create and run a job with custom status events that indicate when an
important event happens, configure one or more runnables to
write a structured log by printing JSON
and define the batch/custom/event field as part of the log.
...
"runnables": [
{
...
"echo '{\"batch/custom/event\":\"EVENT_DESCRIPTION\"}'"
...
}
]
...
"logsPolicy": {
"destination": "CLOUD_LOGGING"
}
...
For example, a job with custom status events that indicate when an important event happens can have a JSON configuration file similar to the following:
{
"taskGroups": [
{
"taskSpec": {
"runnables": [
{
"script": {
"text": "sleep 30; echo '{\"batch/custom/event\": \"EVENT_DESCRIPTION\"}'; sleep 30"
}
}
]
},
"taskCount": 3
}
],
"logsPolicy": {
"destination": "CLOUD_LOGGING"
}
}
Replace EVENT_DESCRIPTION with a description for the
custom status event—for example, halfway done.
After the example job has finished running, the resulting custom status event for each task is similar to the following:
statusEvents:
...
- description: EVENT_DESCRIPTION
eventTime: '...'
type: RUNNABLE_CUSTOM_EVENT
...
What's next
- If you have issues creating or running a job, see Troubleshooting.
- Learn how to view status events.
- Learn how to write task logs.
- Learn about more job creation options.