Write task logs

This document describes how to write task logs and how to create and run a Batch job that has task logs.

When logging is enabled for a job, task logs are generated from messages that the job's runnables print during run time. By configuring your runnables to write task logs, you can surface custom information in Cloud Logging, which can help make your jobs easier to analyze and troubleshoot. To learn more about logs, see Analyze a job using logs.

Before you begin

If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
To get the permissions that you need to create a job that writes logs, ask your administrator to grant you the following IAM roles:
- To create a job:
  - Batch Job Editor (roles/batch.jobsEditor) on the project
  - Service Account User (roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account
- To view logs: Logs Viewer (roles/logging.viewer) on the project
For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create and run a job that has task logs

To create and run a job that you want to have task logs, do the following when you are creating the job:

Enable logs for the job. This allows any logs written for the job to be generated.
For each task log that you want the job to have, add a command that writes a task log to a runnable. When the job runs, a task log gets generated whenever a command to write a task log is executed.

To learn how to write task logs, see Write task logs in this document.

Write task logs

A task log is written for any content that a job's runnables print to the standard output (stdout) stream or standard error (stderr) stream during run time. For example, you can write task logs by using the echo command. The structure of the resulting task log varies based on how you formatted the printed content. Specifically, you can write each task log using one of the following options:

Write an unstructured log by printing a string. Unstructured logs are simple and only let you define the log's textPayload field.
Write a structured log by printing JSON. Structured logs require JSON formatting, which lets you define multiple fields, including some standard log fields, custom fields, and custom status events.

For example, you can use structured task logs to define the log's severity, which you can use as a filter when viewing logs for a job.

Write an unstructured log by printing a string

Unstructured logs let you define a message, which is a string that appears in the log's textPayload field.

To write an unstructured log, print an unformatted string as demonstrated in the following sections.

Example unstructured log

For example, suppose that you want a task log that contains the following string:

MESSAGE

Printing this example string results in a task log that's similar to the following:

insertId: ...
labels: ...
logName: projects/PROJECT_ID/logs/batch_task_logs
receiveTimestamp: ...
resource: ...
severity: INFO
textPayload: MESSAGE
timestamp: ...

Replace the following:

MESSAGE: the message, which is a string that summarizes the purpose of the task log—for example, The summary for a task log..
PROJECT_ID: the project ID of your project.

Print a string

You can print a string using a variety of methods, such as by including the following echo command in a runnable:

echo MESSAGE

For comprehensive examples of jobs that use the echo command to write unstructured task logs, see Create and run a basic job.

Write a structured log by printing a JSON object

Structured logs let you define any of the following:

Standard fields supported by the Cloud Logging agent
Custom fields
Custom status events

To write a structured log, print a JSON object. The following sections demonstrates how to define a log with some of the standard fields and custom fields. If you want to learn how to define a log with custom status events, also see Configure custom status events.

Example structured log

For example, suppose you want a task log that contains the information in the following JSON object, which defines a message, severity, and two custom fields.

{
  "message": "MESSAGE"
  "severity": "SEVERITY"
  "CUSTOM_FIELD_1": CUSTOM_VALUE_1
  "CUSTOM_FIELD_2": CUSTOM_VALUE_2
}

Printing this JSON object results in a task log that's similar to the following:

insertId: ...
jsonPayload:
  "CUSTOM_FIELD_1": CUSTOM_VALUE_1
  "CUSTOM_FIELD_2": CUSTOM_VALUE_2
  message: MESSAGE
labels: ...
logName: projects/PROJECT_ID/logs/batch_task_logs
receiveTimestamp: ...
resource: ...
severity: SEVERITY
timestamp: ...

Replace the following:

MESSAGE: the message, which is a string that summarizes the purpose of the task log—for example, The summary for a task log..
SEVERITY: the severity of the log, which you can use as a filter when viewing logs for a job. The severity must be one of the LogSeverity enums converted to a string with only the first letter capitalized. For example, for the ERROR enum, specify Error.
CUSTOM_FIELD_1 and CUSTOM_FIELD_2: the names of custom fields for the task log—for example, custom_field_1 and custom_field_2.
CUSTOM_VALUE_1 and CUSTOM_VALUE_2: the values of custom fields for the task log, which can be various data types and might need quotes—for example, "the first custom field" and 2.
PROJECT_ID: the project ID of your project.

Print a JSON object

You can print this example JSON object using a variety of methods. For example, the following samples demonstrate some of the possible methods for printing the example JSON object:

Print an equivalent string using the echo command.
Print an equivalent dictionary using Python.

echo command

To print the example JSON object by using the echo command and an equivalent string, include the following command in a runnable:

echo '{\"message\":\"MESSAGE\", \"severity\":\"SEVERITY\", \"CUSTOM_FIELD_1\":CUSTOM_VALUE_1, \"CUSTOM_FIELD_2\":CUSTOM_VALUE_2}'

For example, suppose you create and run a job with the following runnable:

"script": {
  "text": "echo '{\"message\":\"The message for a structured log.\", \"severity\":\"Error\", \"custom_field_1\":\"the first custom field\", \"custom_field_2\":2}'"
}

Then, the resulting task log is similar to the following:

insertId: ...
jsonPayload:
  custom_field_1: the first custom field
  custom_field_2: 2
  message: The summary for a structured task log with error severity.
labels: ...
logName: projects/PROJECT_ID/logs/batch_task_logs
receiveTimestamp: ...
resource: ...
severity: ERROR
timestamp: ...

Python

To print the example JSON object using Python and an equivalent dictionary, include the following sample in a runnable:

#!/usr/bin/env python3

import json

entry = dict(
    severity="SEVERITY",
    message="MESSAGE",
    CUSTOM_FIELD_1=CUSTOM_VALUE_1,
    CUSTOM_FIELD_2=CUSTOM_VALUE_2,
)
print(json.dumps(entry))

For example, suppose you create and run a job with the following runnable:

"script": {
  "text": "#!/usr/bin/env python3\n\nimport json\n\nentry = dict(\nseverity=\"Error\",\nmessage=\"The summary for a structured task log with error severity.\",\ncustom_field_1=\"the first custom field\",\ncustom_field_2=2,\n)\nprint(json.dumps(entry))"
}

Then, the resulting task log is similar to the following:

insertId: ...
jsonPayload:
  custom_field_1: the first custom field
  custom_field_2: 2
  message: The summary for a structured task log with error severity.
labels: ...
logName: projects/PROJECT_ID/logs/batch_task_logs
receiveTimestamp: ...
resource: ...
severity: ERROR
timestamp: ...

What's next

For more information about formatting logs, see Structured logging in the Cloud Logging documentation.
Learn how to write structured task logs that include custom status events.
Learn how to view the task logs for a job.
Learn about more job creation options.