Setting up alerts and notifications

You're viewing Apigee X documentation.
View Apigee Edge documentation.

One of the main purposes of API Monitoring is to keep you informed of unusual events or patterns, such as spikes in traffic or latencies. To track events like these, you can set up alerts, which are triggered when a specified event occurs. An actual event that triggers an alert is called an incident.

You can set specific rules for an alert—for example, it might be triggered when the number of API requests exceeds 3600 over a period of 1 minute. When an incident for the alert occurs, you can view its details in the Events pane of the Cloud Console Alerting dashboard.

If you want to be notified when a particular alert is triggered, you can set up a notification for the alert. When an incident occurs, Apigee sends you a notification with a summary of the event, including links to the Cloud Console and the Apigee UI, for further investigation.

There are two types of alerts in API Monitoring:

Metric-based alerts

Metric-based alerts are triggered when a metric for API data reaches a specified level, or starts to change rapidly. You define an alert rule using two main properties:

  • The metric: The quantity that triggers the alert—for example, number of requests.
  • The threshold: The level that the metric must reach to trigger the alert.

Initial steps for creating a metric-based alert

To create a metric-based alert, start by doing the steps below.

  1. Open the Create alerting policy dashboard in the Google Cloud Console.

    Note: Open the Create alerting policy dashboard in a new tab if you want to follow the steps below.

    Create alerting policy page

  2. Click ADD CONDITION. This opens a new window where you can add the alert condition.

    Configure an alert.

  3. Optional: Add a title for the condition in the Untitled Condition field. We'll leave it blank for this example, in which case the condition is given a default title.
  4. Copy and paste the following in the Find resource type and metric field:

    Apigee proxy (v2)

    The Target pane now appears as shown below.

    Select a metric.

    The pane displays the Select a metric field, where you can specify the metric, such as total traffic or latency percentile.

After completing the initial steps above, you can continue by selecting a metric and adding conditions for the alert in the Configuration pane.

The next two sections present examples that show how to do this:

Traffic spike alert

The following sections show how to create an alert that is triggered when the total API traffic is above 3600 for 1 minute.

Initial steps

To set up the alert, start by doing the initial steps described in the preceding section. When you are done, add the settings described below.

Target settings

In the Target pane:

  1. Copy one of the code samples below, depending on whether you want an alert for requests or responses, and paste it in the Select a metric field.

    Requests

    apigee.googleapis.com/proxyV2/request_count

    Responses

    apigee.googleapis.com/proxyV2/response_count
  2. Click the Select a metric field and select either Apigee proxy request cumulative or Apigee proxy response cumulative.

    Find resource type and metric.

    This displays the fields shown below.

    Select target options.
  3. In the Aggregator field, select sum. This sums the total traffic over the time span specified by Period.
  4. Set Period to 1 minute.

    These settings will trigger an alert based on the total number of requests in each 1 minute interval of the time range.

Configuration settings

The Configuration settings specify the conditions that trigger the alert. The Threshold is the minimum value of the quantity specified in the Target section—which in this case is requests per second—that will set off the alert.

For example, suppose you want the alert to be triggered if the total traffic in any 1 minute interval is above 36. Since the threshold is measured in requests per second, you divide 3600 by 60 to get a threshold of 60.

Enter the following in the Configuration pane:

  • Set Threshold to 60.
  • Set For to 1 minute.

The Configuration pane now appears as shown below.

Configuration settings

Finally, click ADD to add the alert condition. This takes you back to the Create alerting policy dashboard, which displays the new condition, as shown below.

New condition added.

From here you can choose to either:

  • Click ADD Condition to create another condition for the alert, or
  • Click NEXT to go to Step 2, where you will have the option to [create a notification for the alert](#creating-a-notification-for-an-alert), as described in the next section.

If you don't want to create a notification, click NEXT twice without entering anything to bypass Steps 2 and 3.

Click SAVE to save the alert.

Viewing incident details

An API event that triggers an alert is called an incident. You can view the details of recent incidents in the Cloud Console Alerting dashboard.

Metrics and filters

This section describes the metrics and filters you can use to create metric-based alerts.

Metrics

A metric is the quantity that triggers an alert, for example, number of requests. The table below shows the available metrics.

Metric Description
apigee.googleapis.com/proxyv2/request_count Total number of requests received by the Apigee API proxy.
apigee.googleapis.com/proxyv2/response_count Total number of responses returned by the Apigee API proxy.
apigee.googleapis.com/proxyv2/latencies_percentile Percentile of all API proxy responses to a request.

Filters

Filters enable you to create alerts for subsets of API data. A filter specifies a condition that defines the subset. For example, you can create an alert based on data for a single proxy.

You can choose from the following filters:

Filter Description Metrics for which filter can be applied
env Environment for the data. All
location Location for the data. All
proxy_name Proxy for the data. All
runtime_version Runtime version for the data. All
instance_id Instance ID for the data. All
method HTTP method for the data. All
response_code HTTP response code for the data. apigee.googleapis.com/proxyv2/response_count
fault_code Fault code for the data. apigee.googleapis.com/proxyv2/response_count
fault_source Fault source for the data. apigee.googleapis.com/proxyv2/response_count
percentile Apigee proxy response latencies percentile. apigee.googleapis.com/proxyv2/latencies_percentile

Log-based alerts

In addition to the metric-based alerts described in the previous section, you can also create log-based alerts, which monitor API data stored by Cloud logging.

Setting up a log-based alert

The next example shows how to set up a log-based alert.

  1. Open the Google Cloud Console in your browser.
  2. In the upper-left corner of the Console, open the navigation menu and select Logging > Logs Explorer.
  3. In the Query pane, enter a query like the example shown below.
    resource.type="apigee.googleapis.com/Environment"
    logName="projects/<project_id>/logs/apigee.googleapis.com%2Fanomaly_event"

    Set time between alerts.

    See Create a log-based alert (Logs Explorer) for more information.

  4. Click Run query. The results are displayed in the Verify that the query returns the log entries you want to alert on.
  5. Click Create alert below the Query pane.

    Create a new query.
  6. Specify a name and a description of the alert. Note that the Alert Description field supports Markdown formatting. Once you have filled in the fields, select Next to continue.

    Alert details
  7. In the Choose logs to include in the alert pane, verify that the entries below Define log entries to alert on match the query you entered in Step 3.

    Choose logs to include dialog.

    Select Next to continue.

  8. Next, specify the minimum amount of time that you want to elapse between alerts for this policy. For example in the example policy below, if two log entries matching the filter are written within a 10 minute period, you will receive an alert for the first log entry that was written, but no alert for the second.

    Set time between alerts.
  9. Finally, if you want to receive a notification when an alert occurs, specify a recipient in the Notification Channels field, and then click Save.

    Set who to notify.

Creating a notification for an alert

If you want to be notified immediately when an incident occurs, you can create a notification for the alert. When the alert is triggered, Apigee sends you a notification. You can choose any of the following channels to receive the notification:

  • Email
  • PagerDuty
  • Slack
  • Webhooks

To create a notification for an alert:

  1. If you just created the alert (as shown in the preceding example) and are currently viewing the Create alerting policy dashboard, skip to the next step.

    Otherwise, open the Policies pane in the Cloud Console Alerting dashboard, and in the row for the alert you created, click the three dots icon at the end of the row and select Edit.

    Notification options

    This opens the Edit alerting policy dashboard:

    Notification options
  2. In the What do you want to track? section, click NEXT.

    Notification options
  3. In the Who should be notified section, select Notification Channels, where you can choose from existing channels for receiving notifications.

    If you want to create a new notification channel, select MANAGE NOTIFICATION CHANNELS. This opens the Notification channels dashboard in the Cloud Console, where you can add channels. See Managing notification channels for more details.

  4. Click NEXT. This displays the options shown below.

    Notification options
  5. In the Alert name field, enter a name for the alert.
  6. In the Documentation field, enter the URL for a web page that provides information on how to fix the issue that caused the alert. The URL will be included in the notification.
  7. Click Save to create the alert.

When an alert is triggered, you will receive a notification providing a summary of the incident and when it occurred. The notification also contains two links to help you investigate the incident:

Latency alert

The following section shows how to create an alert that is triggered when the 90th percentile of API latency is above 600 ms for 10 minutes.

Initial steps

To create the alert, start by doing the Initial steps described in the first section. After doing so, the Target pane appears as shown below.

Select a metric.

Target settings

Next, do the following steps in the Target pane:

  1. Copy the code below and paste it in the Select a metric field.
    apigee.googleapis.com/ProxyV2/latencies-percentile

    or select Percentile of Apigee proxy response.

    Select latency metric.

  2. Under Filter, click in the Add a filter field and select percentile.

    Select a metric.

  3. In the Value field that appears below Select a metric, select 90.

    Select a metric.

  4. Click Apply.

Configuration settings

In the Configuration pane, do the following steps:

  1. Set Threshold to 600.

    Select threshold.

With these settings, an alert will be triggered when the 90th percentile of the API's latency is above 600 for 10 minutes.

The UI displays the latency graph with the threshold, as shown below.

Select a metric.

To create the alert, click Add at the bottom of the UI. This takes you back to the Create alerting policy dashboard.

See Creating a notification to learn how to create a notification for the alert.