Shape the future of software operations and make your voice heard by taking the 2023 State of DevOps survey.

Annotate alerts with user-defined documentation

This page describes the content you can include in the documentation field of an alerting policy. This field supports plain text, Markdown, variables, and channel-specific controls.

Using Markdown

The documentation field supports the following subset of Markdown tagging:

  • Headers, indicated by initial hash characters.
  • Unordered lists, indicated by initial plus, minus, or asterisk characters.
  • Ordered lists, indicated by an initial number followed by a period.
  • Italic text, indicated by single underscores or asterisks around a phrase.
  • Bold text, indicated by double underscores or asterisks around a phrase.
  • Links, indicated by [link text](url) syntax.

For more information about this tagging, see any Markdown reference, for example, Markdown guide.

Using variables

To tailor the content of your documentation, you can use variables of the form ${varname}. When the documentation is sent with a notification, the string ${varname} is replaced with a value drawn from the corresponding Google Cloud resource, as described in the following table. The following screenshot shows the documentation included in an email notification, created from the documentation template specified for an alerting policy:

Documentation included in email.

For information on creating a documentation template for an alerting policy, see the optional step for specifying the documentation to be included in notifications.

The following variables are available for use in documentation fields:

Variable Value The REST resource name of the condition, such as
condition.display_name The display name of a condition, such as CPU usage increasing rapidly.
log.extracted_label.KEY The value of the label KEY, extracted from a log entry. For log-based alerts only; for more information, see Create a log-based alert (Monitoring API).
metadata.system_label.KEY The value of the system-supplied resource metadata label KEY.1
metadata.user_label.KEY The value of the user-defined resource metadata label KEY.1,3
metric.type The metric type, such as
metric.display_name The display name for the metric type, such as CPU utilization.

The value of the metric label KEY.1
To find the labels associated with the metric type, see Metric list.

When the value of the variable ${metric.label.KEY} doesn't start with a digit, a letter, a forward slash (/), or an equal sign (=), Monitoring omits the label from notifications. The REST resource name of the policy, such as projects/foo/alertPolicies/1234.
policy.display_name The display name of a policy, such as High CPU rate of change.
policy.user_label.KEY The value of the user label KEY.1

Keys must start with a lowercase letter. Keys and values can contain only lowercase letters, digits, underscores, and dashes.

project The ID of the scoping project of a metrics scope, such as a-gcp-project.
resource.type The monitored-resource type, such as gce_instance.
resource.project The project ID of the monitored resource of the alerting policy.
resource.label.KEY The value of the resource label KEY.1,2,3
To find the labels associated with the monitored-resource type, see Resource list.

1 For example, ${} is replaced with the value of the zone label. The values of these variables are subject to grouping; see null values for more information.
2 To retrieve the value of the project_id label on a monitored resource in the alerting policy, use ${resource.project}.
3 You can't access user-defined resource metadata labels by using resource.label.KEY. Use metadata.user_label.KEY instead.

null values

Values for the metric.*, resource.* and metadata.* variables are derived from time series. Their values can be null if no values are returned from the time series query.

  • The resource.label.KEY and metric.label.KEY variables can have null values if your alerting policy uses cross-series aggregation (reduction), for example, calculating the SUM across each of the time-series that match a filter). When using cross-series aggregation, any labels not used in grouping are dropped and as a result they render as null when the variable is replaced with its value. All labels are retained when there is no cross-series aggregation.
  • Values for metadata.* variables are available only if the labels are explicitly included in a condition's filter or grouping for cross-series aggregation. That is, you must refer to the metadata label in either filter or grouping for it to have a value for the template.

Other usage notes

  • Only the variables in the table are supported. You cannot combine them into more complex expressions, like ${varname1 + varname2}.
  • To include the literal string ${ in your documentation, escape the $ symbol with a second $ symbol, and $${ renders as ${ in your documentation.
  • These variables are replaced by their values only in notifications sent through notification channels. In the Google Cloud console, when the documentation is shown, you see the variables, not the values. Examples in the console include the descriptions of incidents and the preview of the documentation when creating an alerting policy.

Using channel controls

The text in the documentation field can also include special characters used by the notification channel itself to control formatting and notifications.

For example, Slack uses @ for mentions. You can use this to link the notification to a specific user ID. Suppose you include a string like this in the documentation field:

<@backendoncall> policy ${policy.display_name} triggered an incident

When the documentation field is received by the relevant Slack channel as part of the notification, this line triggers an additional message to the user ID backendoncall that, for example, the policy High CPU rate of change triggered an incident. The mention must refer to a user ID, not a name.

These additional options are specific to the channels; for more information on what may be available, consult the documentation provided by the channel vendor.