New ways to manage and automate your Stackdriver alerting policies
Software Engineer, Google Stackdriver
Product Manager, Google Stackdriver
If your organization uses Google Stackdriver, our hybrid monitoring, logging and diagnostics suite, you’re most likely familiar with Stackdriver alerting. DevOps teams use alerting to monitor and respond to incidents impacting their applications running in the cloud. We’ve received a lot of great feedback about the Stackdriver alerting functionality, notably, the need for a programmatic interface to manage alerting policies and a means of automating them across different cloud projects.
Today, we're pleased to announce the beta release of new endpoints in the Stackdriver Monitoring v3 API to manage alerting policies and notification channels. Now, it’s possible to create, read, write, and manage your Stackdriver alerting policies and notification channels. You can perform these operations using client libraries in one of the supported languages (Java or C#, with more to come later) or by directly invoking the API, which supports both gRPC and HTTP / JSON REST protocols. There's also command line support in the Google Cloud SDK via the gcloud alpha monitoring policies, gcloud alpha monitoring channel-descriptors, and gcloud alpha monitoring channels commands.
Providing programmatic access to alerting policies and notification channels can help automate common tasks such as:
- Copying policies and notification channels between different projects, for example between test, dev and production
- Disabling and later re-enabling policies and notification channels in the event of alerting storms
- Utilizing user labels to organize and filter notification channels and policies
- Programatically verifying SMS channels as new SMS numbers get added to the team
If you have multiple alerting policies configured by various teams within a single Google Cloud project, navigating and organizing these policies can be challenging. With the Stackdriver Alerting API, you can add "user labels" to annotate policies with metadata, which then makes it easier to find and navigate these policies. For example, here’s how to list all your policies:
Here’s how to tag a given policy with your team name:
You can then easily find policies that have your team name:
When someone new joins your DevOps team, it can be a very tedious process to update all your policies so that they receive all the relevant notifications. Now, with the Alerting API, you can quickly add your new teammate to all of the alerting policies that your team owns.
First, find the channels that belong to the team member:
If they don't already have a notification channel, you can create one:
Then, add a notification channel to a given policy:
Combined with the policies list command, adding the notification channel to all of your team's policies is a matter of a simple BASH script, not tons of tedious point-and-click configuration.
Disabling alerts to a given endpoint
If you're in the middle of a pagerstorm and getting endless alerts, it’s easy to disable notifications to a channel without removing that channel from all existing policies:
To summarize, the alerting policy and notification channel management features in the Monitoring v3 API will help you simplify and automate a number of tasks. We hope that this saves you time, and we look forward to your feedback!
Please send your feedback to google-stackdriver-discussion_AT_googlegroups.com.