This document illustrates using the Cloud Monitoring API to create, edit, delete, list, and get metric-based alerting policies programmatically. The examples show how to use of the Google Cloud CLI and how to use client libraries. This content does not apply to log-based alerting policies. For information about log-based alerting policies, see Monitoring your logs.
These tasks can also be performed by using the Google Cloud console; for more information, see the following documents:
- Create metric-threshold alerting policies by using the Google Cloud console
- Manage alerting policies by using the Google Cloud console
About alerting policies
An alerting policy is represented by an AlertPolicy
object,
which describes a set of conditions indicating a potentially
unhealthy status in your system. Alerting policies reference
notification channels, which let you specify how you want to be informed
that an alerting policy has been triggered.
Each alerting policy belongs to a scoping project of a metrics scope. Each
project can contain up to 500 policies.
For API calls, you must provide a “project ID”; use the
ID of the scoping project of a metrics scope as the value. In these examples,
the ID of the scoping project of a metrics scope is a-gcp-project
.
The AlertPolicy
resource supports five operations:
- Creating new policies
- Deleting existing policies
- Retrieving specific policies
- Retrieving all policies
- Modifying existing policies
Alerting policies can be expressed in JSON or YAML, which lets you record policies in files, and use files to back up and restore policies. With Google Cloud CLI, you can create policies from files in either format. With the REST API, you can create policies from JSON files. See Sample policies for a selection of alerting policies in JSON format.
The following examples use the gcloud
interface and the API
to illustrate these basic use cases. The API samples are excerpted
from a sample program that uses the API to implement a backup and restore
system for alerting policies. Fuller samples are shown in Example: backup
and restore.
Before you begin
Before writing code against the API, you should:
- Be familiar with the general concepts and terminology used with alerting policies; see Alerting overview for more information.
- Ensure that the Cloud Monitoring API is enabled for use; see Enabling the API for more information.
- If you plan to use client libraries, then install the libraries for the languages that you want to use; see Client Libraries for details. Currently, API support for alerting is available only for C#, Go, Java, Node.js, and Python.
If you plan to use the Google Cloud CLI, then install it. However, if you use Cloud Shell, then Google Cloud CLI is already installed.
Examples using the
gcloud
interface are also provided here. Note that thegcloud
examples all assume that the current project has already been set as the target (gcloud config set project [PROJECT_ID]
) so invocations omit the explicit--project
flag. The ID of the current project in the examples isa-gcp-project
.
-
To get the permissions that you need to create and modify alerting policies by using the Cloud Monitoring API, ask your administrator to grant you the Monitoring AlertPolicy Editor (
roles/monitoring.alertPolicyEditor
) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.You might also be able to get the required permissions through custom roles or other predefined roles.
For detailed information about IAM roles for Monitoring, see Control access with Identity and Access Management.
Design your application to single-thread Cloud Monitoring API calls that modify the state of an alerting policy in a Google Cloud project. For example, single-thread API calls that create, update, or delete an alerting policy.
Create an alerting policy
To create an alerting policy in a project, use the
alertPolicies.create
method. For information about how to invoke this
method, its parameters, and the response data, see the reference page
alertPolicies.create
.
You can create policies from JSON or YAML files.
The Google Cloud CLI accepts these files as arguments, and
you can programmatically read JSON files, convert them to AlertPolicy
objects, and create policies from them
by using the alertPolicies.create
method. If you
have a Prometheus JSON or YAML configuration file with an alerting rule, then
the gcloud CLI can migrate it to a Cloud Monitoring alerting
policy with a PromQL condition. For more information, see
Migrate alerting rules and receivers from Prometheus.
The following samples illustrate the creation of alerting policies, but they don't describe how to create a JSON or YAML file that describes an alerting policy. Instead, the samples assume that a JSON-formatted file exists and they illustrate how to issue the API call. For example JSON files, see Sample policies. For general information about monitoring ratios of metrics, see Ratios of metrics.
gcloud
To create an alerting policy in a project, use the gcloud alpha monitoring
policies create
command. The following example creates an alerting policy in
a-gcp-project
from the rising-cpu-usage.json
file:
gcloud alpha monitoring policies create --policy-from-file="rising-cpu-usage.json"
If successful, this command returns the name of the new policy, for example:
Created alert policy [projects/a-gcp-project/alertPolicies/12669073143329903307].
The file rising-cpu-usage.json
file contains the JSON for a policy with
the display name “High CPU rate of change”. For details about this policy, see
Rate-of-change policy.
See the
gcloud alpha monitoring policies create
reference for more information.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
The created AlertPolicy
object will have additional fields.
The policy itself will have name
, creationRecord
, and mutationRecord
fields. Additionally, each condition in the policy is also given a name
.
These fields cannot be modified externally, so there is no need to set them
when creating a policy. None of the JSON examples used for creating
policies include them, but if policies created from them are retrieved after
creation, the fields will be present.
List and get alerting policies
To retrieve a list of the policies in a project, use the
alertPolicies.list
method.
Use this method to retrieve policies and apply some action to each of
them, for example, backing them up. This method also supports
filter
and orderBy
options to restrict and sort the results;
see Sorting and Filtering.
If you are looking for a specific policy and you know its name,
you can use the alertPolicies.get
method
to retrieve only that policy. The name of a policy is the value of the name
field, not the displayName
, in the AlertPolicy
object. The name of
a policy has the format projects/[PROJECT_ID]/alertPolicies/[POLICY_ID]
,
for example:
projects/a-gcp-project/alertPolicies/12669073143329903307
gcloud
To list all alerting policies in a project, use the gcloud alpha monitoring
policies list
command:
gcloud alpha monitoring policies list
If successful, the list
command provides a listing of all the policies in the
specified project, formatted as YAML. For example, the
policy with the display name “High CPU rate of change”
in the project a-gcp-project
is listed like this,
among the other policies listed:
---
combiner: OR
conditions:
- conditionThreshold:
aggregations:
- alignmentPeriod: 900s
perSeriesAligner: ALIGN_PERCENT_CHANGE
comparison: COMPARISON_GT
duration: 180s
filter: metric.type="compute.googleapis.com/instance/cpu/utilization" AND resource.type="gce_instance"
thresholdValue: 0.5
trigger:
count: 1
displayName: CPU usage is increasing at a high rate
name: projects/a-gcp-project/alertPolicies/12669073143329903307/conditions/12669073143329903008
creationRecord:
mutateTime: '2018-03-26T18:52:39.363601689Z'
mutatedBy: [USER@DOMAIN]
displayName: High CPU rate of change
enabled: true
mutationRecord:
mutateTime: '2018-03-26T18:52:39.363601689Z'
mutatedBy: [USER@DOMAIN]
name: projects/a-gcp-project/alertPolicies/12669073143329903307
---
To list a single alerting policy, use gcloud alpha monitoring policies
describe
, instead, and specify the name of the policy. For example, this
command returns only the listing above:
gcloud alpha monitoring policies describe projects/a-gcp-project/alertPolicies/12669073143329903307
See the gcloud alpha monitoring policies list
and describe
references for more information. The describe
command corresponds to the
alertPolicies.get
method in the API.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Delete an alerting policy
To delete a policy from a project, use the
alertPolicies.delete
method and supply the name of the
alerting policy to delete.
gcloud
To delete an alerting policy, use gcloud alpha monitoring policies
delete
, and specify the name of the policy to delete. For example, the
following command deletes the
policy with the display name “High CPU rate of change”:
gcloud alpha monitoring policies delete projects/a-gcp-project/alertPolicies/12669073143329903307
See the
gcloud alpha monitoring policies delete
reference for more information.
Modify an alerting policy
To modify an alerting policy, use the
alertPolicies.patch
method (in the REST API).
Other API implementations and the gcloud
interface call this update
instead of patch
.
An update operation can wholly replace the existing policy, or it can modify
a subset of fields. An update operation takes a new AlertPolicy
object and
an optional field mask.
If a field mask is specified, then any field listed in the field mask is updated with the value in the supplied policy. If the supplied policy does not include a field mentioned in the field mask, then that field is cleared and set to its default value. Any field not listed in the mask retains its previous value.
If no field mask is specified, then the existing policy is replaced by the
supplied one, but the name (projects/[PROJECT_ID]/alertPolicies/[POLICY_ID]
)
is reused. Any conditions in the new policy that have name
values that include
a CONDITION_ID
will keep those names. If not, new condition and policy names
are created.
When using the gcloud
command-line to update policies, command-line flags
rather than a field mask are used to specify fields to update.
See the gcloud alpha monitoring policies update
for details.
Enable or disable an alerting policy
To enable or disable a policy, change the value of the boolean
enabled
field in the AlertPolicy
object. Note that after you enable
a policy, it can still be triggered by data collected while it was disabled.
gcloud
To disable an alerting policy, use the
gcloud alpha monitoring policies update
command and provide the --no-enabled
flag. The following command disables the “High CPU rate of change”
alerting policy in the project a-gcp-project
:
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 --no-enabled
To enable the policy, use the same command and provide the --enabled
flag.
See the gcloud alpha monitoring policies update
reference for more information. The update
command corresponds to the
alertPolicies.patch
method in the REST API.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Update notification channels in an alerting policy
You can also update the notification channels referenced by an alerting policy. Alerting policies refer to notification channels by name. The channels must exist before they can be used in an alerting policy.
You create and manage notification channels programmatically by using
the NotificationChannel
and
NotificationChannelDescriptors
resources.
These examples in this
section assume these channels already exist, and uses of these APIs also
appear in the programmatic samples.
For more discussion of the notification-channel objects, see Create and manage notification channels by API.
gcloud
To modify the notification channels in an alerting policy, use the
gcloud alpha monitoring policies update
command. There are several
flags related to notification channels, letting you remove notification
channels, replace notification channels, and add new notification channels.
For example, the policy with the display name “High CPU rate of change” in the project a-gcp-project was created with no notification channels.
To add a notification channel to this policy, use the gcloud alpha monitoring
policies update
command, and specify the channel to add with the
--add-notification-channels
flag:
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 \
--add-notification-channels="projects/a-gcp-project/notificationChannels/1355376463305411567"
See the gcloud alpha monitoring policies update
reference for more information. The update
command corresponds to the
alertPolicies.patch
method in the REST API.
The notification channel added here must already exist; see Create a notification channel for more information.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Modify documentation in an alerting policy
A policy can include documentation that is included with incidents and notifications associated with the policy. Use this field to include information to help responders understand and handle the problem indicated by the alert policy. Documentation is included in email notifications and notification types that allow for it; other channel types may omit it.
gcloud
To add documentation to a policy, or replace existing documentation, use thegcloud alpha monitoring policies update
command and provide the
--documentation-format="text/markdown"
flag (the only supported format)
and either the --documentation
flag (to enter the value from the command line)
or the --documentation-from-file
flag (to read the value from a file).
For example, the policy with the display name “High CPU rate of change” in the project a-gcp-project was created with no documentation.
The following command sets the documentation
field in the specified policy to
the contents of the cpu-usage-doc.md
file:
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 \
--documentation-format="text/markdown" \
--documentation-from-file="cpu-usage-doc.md"
See the gcloud alpha monitoring policies update
reference for more information. The update
command corresponds to the
alertPolicies.patch
method in the REST API.
Add an alerting policy to a dashboard
To display a summary of a single-condition alerting policy on custom dashboard,
add an AlertChart
widget to the dashboard.
You use the
dashboards.create
method for a new dashboard and the
dashboards.patch
method for an existing dashboard.
If you specify a multi-condition alerting policy, then
the AlertChart
widget won't display data.
For detailed information about using these API methods, see Create and manage dashboards by API.
Example: backup and restore
All of the API examples shown are pulled from a larger application that can back up the alerting policies in a project to a file and can restore the policies, possibly to another project. If the projects used for backup and restore are different, the application effectively exports and imports policies from one project to another.
This section shows the code for backup and restore in context rather than as a set of small, isolated excerpts.
Backing up policies
The back-up operation is straight-forward. The set of alerting policies and the set of notification channels in each project is collected and saved to external storage in JSON.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Restoring the backed-up policies
The restoration process is more complex than the original back-up. You can restore to the project you originally backed up. You can also restore to a different project, effectively providing import of alerting policies.
If restoring to the same project, any existing channels or policies are updated if they still exist. If they don't, they are recreated. Read-only fields, like the creation and mutation records, in the backed-up policies are cleared by the restoration process before policies and notifications are recreated.
You can use a policy saved in one project to create a new, or similar, policy in another project. However, you must first make the following changes in a copy of the saved policy:
- Remove the following fields from any notification channels:
name
verificationStatus
- Create notification channels before referring to the channels in alerting policies (you need the new channel identifiers).
- Remove the following fields from any alerting policies you are recreating:
name
condition.name
creationRecord
mutationRecord
If the policy is being recreated in a new project, the names of any conditions in the backed-up policies are cleared along with the creation and mutation records.
Additionally, when a notification channel is recreated in a different project, it gets a different name, so the restore process has to map the names of channels in backed-up alerting policies to their new names, and replace the old names with the new ones.
In addition to the names of notification channels, the value of the
verificationStatus
field cannot be set when the channel is created or
updated, so a sentinel value, unspecified
is used. After channels have
been restored into a new project, they must be explicitly verified.
C#
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To authenticate to Monitoring, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Alerting and Google Cloud CLI
In Google Cloud CLI, the command group for managing alerting policies and
notification channels is monitoring
, which is in alpha release.
The monitoring
group is available in the alpha
component.
That is, these commands will all begin with:
gcloud alpha monitoring
To check if you have the alpha
component installed, run this command:
gcloud components list
If you don't have the alpha
component installed, run this command to
install it:
gcloud components install alpha
If you do have the alpha
component, check for the monitoring
group by
running this command:
gcloud alpha monitoring --help
If the monitoring
group is not included, Google Cloud CLI will prompt you to
add it:
You do not currently have this command group installed.
[...]
Do you want to continue (Y/n)? y