Create PromQL alerts in Cloud Monitoring now in Public Preview

Last year, we introduced Managed Service for Prometheus so that you can scale your Prometheus environment more easily that hosting it yourself. As part of that announcement, we also added support for Prometheus’ popular PromQL query language in Cloud Monitoring. Since then, we’ve heard from experienced and new Prometheus users that they’d like to manage alerts in a single ecosystem. Today, we’re excited to announce that Cloud Monitoring now supports alerting using PromQL in Public Preview. You can now create globally scoped alerting policies based on PromQL queries alongside your Cloud Monitoring metrics and dashboards, without having to maintain backend services. 

In this release, you can:

• Write globally scoped PromQL-based alerting policies in Cloud Monitoring 

• Reference Prometheus, GCP system, and custom metrics in your alerting policy 

• Route the notification to any Cloud Monitoring-supported notification channel. Use Email, Slack, SMS, and Mobile push to send the notification to your team members. Use Webhooks to send the notification to any public endpoint or Pub/Sub for any private endpoint. 

• Customize the subject line in an Email notification channel 

• Easily migrate your existing Prometheus alert_rules to Cloud Monitoring 

• Manage your configuration with Terraform 

If you already have Prometheus alert rules, then you can migrate them to Cloud Monitoring alerting policies containing a PromQL query. You can also create your own PromQL alerting policies directly in Cloud Monitoring by using the Monitoring API or gCloud CLI.

Understanding alerting policies 

The following Prometheus alert rule triggers if your Kuberentes volume is at 90% of the available disk space. Let’s review a few of its fields:

Alert: Sets a name for the alert to help users identify what’s happening

Expr: The PromQL expression to evaluate. In this case, if the volume’s capacity exceeds 90%, then an alert will fire.  

For: Specifies the length of time during which each evaluation of the query must generate a `true` value before the alert fires.

Summary: The customized subject line to be used in the alert.

The labels and annotations fields provide additional information about the alert and can be used for passing additional context or actions.

Many Prometheus alert rules contain Prometheus templating language constructs like {{ $value}}, {{$labels}}, and {{$labels.<name>}}. For the time being these constructs will be rendered verbatim if you include them in your summary or description.

The following Prometheus alert rule will be rendered correctly now and in the future, because we replace the Prometheus templating language constructs with the equivalent document variables:

Migrate alert rules to Cloud Monitoring 

For most Prometheus users, alert rules are stored in a Prometheus config file or rules file. You can now migrate these files to Cloud Monitoring alerting policies with PromQL queries. This way, you can store all of your alert policies in one place. Let’s look at a sample migration process:

The following PromQL expression sends an alert if any instance latency spikes above the 95th percentile during the weekdays:

The following Prometheus rules file sample shows the latency evaluation query and several other important fields:

alerting_rules.yaml

You can migrate this alert rule by providing it to the gCloud migration command:

Cloud Monitoring then creates an alerting policy that contains the PromQL query:

These examples show how easy it is to get started with Cloud Monitoring’s new PromQL Alerting capabilities. You can migrate your existing Prometheus alert rules to Cloud Monitoring alert policies, or create PromQL alerting policies in Cloud Monitoring with the API or CLI.  

If you want to install alerts locally in your clusters and have your clusters route them to a Prometheus Alert Manager, then you can continue to use the Managed Service for Prometheus rule evaluation options for managed or self-deployed collection. Otherwise, you can reduce the time you spend on alert management by migrating your existing Prometheus alert rules to Cloud Monitoring alerting policies.

To learn more, check out our documentation:

• For a general overview of PromQL Alerting, including a list of migration options and alerting rule-to-alerting policy field mapping, see Alerting policies with PromQL.

• For detailed information about how to migrate alerting rules and receivers with the Google Cloud CLI, see Migrate alerting rules and receivers from Prometheus.

• For a walkthrough of how to use Cloud Monitoring API to create alerting policies with a PromQL query, including several examples, see Create alerting policies with a PromQL query.

As always, please leave use feedback during the preview so we can improve the experience!