Abuse detection

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

Preview — Advanced API Security Abuse Detection

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features may have limited support, and changes to pre-GA features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

Support for the Preview launch of Advanced API Security Abuse Detection is available through the Apigee - Google Cloud Community forum.

Advanced API Security's abuse detection lets you view security incidents involving your APIs. A security incident is a group of events with similar patterns that could represent a security threat. Advanced API Security uses machine learning models to detect patterns that are a sign of malicious activity, including API scraping and anomalies, and cluster events together based on similar patterns.

When Advanced API Security detects a security incident, it reports the following:

• The risk level and duration of the incident
• The proxies affected by the incident
• The IP addresses of the incident events
• The detection rules that were triggered by the incident
• The countries of origin of the incident

and other related information about the incident.

You can access abuse detection either through the Apigee UI, as described below, or through the Incidents API or the Security stats API

See Required roles for abuse detection for the roles needed to perform abuse detection tasks.

Help improve your machine learning models for abuse detection

Apigee requests your help in improving the machine learning models for abuse detection in your organization, by allowing us to train the models on your data. Training the models on your data helps improve their accuracy for detecting security incidents. The training will only apply to your models, and they won't be shared with any other Google Cloud customers.

When you first open the Abuse detection page in the Apigee UI, you will see a banner that requests your permission to train your organization's security models on your data.

To use this feature, you must enable the add-on. If you are a Subscription customer, you can enable the add-on for your organization. See Manage Advanced API Security for Subscription organizations for more details. If you are a Pay-as-you-go customer, you can enable the add-on in your eligible environments. For more information, see Manage the Advanced API Security add-on.

Open the Abuse detection page

To open the Abuse detection page:

• If you are using the https://console.cloud.google.com/apigee: Select Advanced API security > Abuse detection.
• If you are using the classic Apigee UI: Select Analyze > API Security > Abuse detection.

This displays the main Abuse detection page:

Note: When you first open the Abuse detection page, you will see a banner that requests permission to train your organization's security models on your data. If you agree, click Enable improvement of my Apigee organization's Advanced API Security models below the banner.

Change permissions for allowing Apigee to improve your machine learning models

You can change your permissions for allowing Apigee to improve your machine learning models at any time, by clicking Settings at the top right of the Abuse detection page and selecting the option to either enable or disable this feature.

Main Abuse detection page

At the top of the page, you can select one of the following recent time periods in which to view incidents: the past 12 hours, 1 day, 1 week, or 2 weeks.

The table in the page displays the environments in your organization that are affected by security incidents during the selected time interval.

Note: If no environments are displayed, it means that no incidents have been detected in any environment in the selected time interval. In this case, you can try selecting a longer time interval, which can be up to 14 days in the past.

Each row of the table also displays the following:

• Environment: The environment in which the abuse occurred.
• Total incidents: The total number of incidents in the environment during the selected time interval. See Limitations on incidents and data displayed for more information on what incidents and data are displayed in the UI.

• Risk level: Shows the number of incidents at three risk levels: severe, moderate, and low. Risk level is based on different characteristics of an incident, such as the number of rules detected, their types, and the relative size of the incident compared to legitimate traffic. The risk level is intended to help you prioritize which incidents to investigate, so you can focus on the most critical ones.

  Risk level can be one of the following:

  • Severe: Severe incidents present high risk. We recommend that you prioritize investigating them.
  • Moderate: Moderate incidents present some risk, but less than those with severe risk, and we recommend that you prioritize them over low risk incidents.
  • Low: Low risk incidents can be investigated last, after you have investigated the higher risk incidents.

  The number next to each risk level indicates the the number of incidents at that risk level.

Environment details

To view the incidents in an environment for the selected time range, select the environment in the table shown above. This opens the Environment details view:

If you see an incident or detected traffic, and want to create a security action to block or flag requests related to the incident or detected traffic, click Create Security Action at the top of the page. This opens the Security actions page.

Limitations on incidents and data displayed:

• The Abuse detection UI does not display incidents whose Last event detected is over 14 days in the past.
• The UI only only displays incident data received after 14 days in the past.

To get additional information about these incidents, you can do either of the following:

1. Call the Incidents API to get a list of incidents outside the 14-day range.
2. Call the Security reports API to gather more details about those incidents.

The Environment details view has two tabs:

• Incidents: Shows a list of incidents in the environment and information about them.
• Detected traffic: Shows details of detected abuse traffic related to the incidents.

Incidents

The Incidents tab of the Environment details view, shown above, displays the following options:

• Environment: Change the environment in which to view incidents.
• Proxy: You can select All to display incidents for all proxies, or an individual proxy to only display incidents for the selected proxy.
• Include archived incidents: When this is selected, the incidents list displays archived incidents. Archived incidents are displayed with an icon next to them:

  To hide archived incidents from the list, deselect Include archived incidents. You might want to hide archived incidents if there are many incidents displayed and you don't want to view them all, or if you want to hide incidents that you have already investigated.

The Incidents view also displays the following:

• Incident name: A generated name summarizing the incident.
• Risk level: The risk level of the incident.

• Detection rules: A list of detection rules that were triggered by the incident.

  Note: An incident can be triggered by multiple detection rules, in which case all the detection rules that triggered the incident are listed.
• Incident traffic: The total number of events: API calls that were tagged by one of the detection rules related to the incident.
• First event detected: The date and time the first event in the incident was detected.
• Last event detected: The date and time the last event in the incident was detected.
• Duration: The duration of the incident, from its first event to its last.
• UUID: The universally unique identifier of the incident.

Incident details

To view the details of an incident, click its name in the table. This displays the Overview pane of the Incident details view, as shown below:

As in the Environment details view, you can click Create Security Action at the top of the page to create a security action in response to the incident.

The Incident details view has two tabs, Overview and Attributes. Attributes—also known as dimensions—are groupings of the data that let you view the incident in different ways. For example, the API products attribute lets you view the incident data by API product.

The Overview and Attributes tabs are described below.

Archive incidents

To help you distinguish between the incidents you have already investigated from those that you haven't, you can archive the incidents that no longer require your attention. Archiving an incident hides it from the Environment details > Incidents list (provided that Include archived incidents is not selected). Archiving does not delete an incident: you can always unarchive it if you change your mind.

To archive an incident, select Archive at the top of the Incident details view. After you do so, the Archive button label changes to Unarchive.

Unarchive incidents

To unarchive an archived incident:

1. In the Environment details > Incidents view, click the icon next to the incident you want to unarchive:
2. At the top of the incident list, click Unarchive.

Alternatively, if you are in the Incident details view for the incident, click Unarchive.

Overview and Attributes tabs

Overview

The Overview pane displays basic information about the incident, including the following:

• Incident name: The name of the incident.
• Risk level: The risk level of the incident.
• Impacted proxies: The number of proxies affected by the incident. Click View proxies to view the affected proxies.
• Duration: The duration of the incident, from its first event to its last.
• Events: Displays a time series graph of events in the incident. For each time point in the graph, the corresponding y value is the total number of events in a short time period around that time. If you hover the cursor over a point in the graph, the number of events in the most recent time period is displayed below Value. You can see the values where the time periods change by moving the cursor to the left or right and observing where the values change.

  The Events pane also displays the following information about events:

  • First event detected: The date and time the first event in the incident was detected.
  • Last event detected: The date and time the last event in the incident was detected.
  • Total incident traffic: The total number of events related to the incident.

  To view the IP addresses related to the incident, click View All IP Addresses.

  Note: This displays unique IP addresses, even if more than one incident corresponds to the same IP address.
• Top rules detected: Displays up to five of the top groups of rules detected, including the following information:
  • Dominant rules: The most significant detection rules that were triggered by the incident.
  • Dominant rules API events: The number of API events tagged by the dominant rules.
  • Total rules detected: The number of detection rules that were triggered by the incident.

  To view all rules, click View All Rules at the bottom of the card.

• Top countries detected: A map showing the countries that were sources of events in the incident. Below the map is a chart that shows up to five of these countries and the percentage of the total traffic originating in these countries.

  Note: If the country of origin of the events can't be determined, the map displays not set.

  To view all countries, click View All Countries at the bottom of the card.

Attributes

The Attributes view lets you drill down on the details of an incident. Attributes—also known as dimensions— are groupings of the data that let you view the incident in different ways. For example, the API products attribute lets you view the incident data by API product.

To view the Attributes, select Attributes at the top of the Incident details view.

The left pane displays all of the attributes and the number of distinct values for each attribute. You can select an attribute to view its incident details.

The picture above shows the Attributes view with API Products selected. The API Products pane displays graphs of the percentage of API calls made for each API product. See What does it mean when an attribute has the value (not set) for information about the (not set) value.

The Filter field lets you filter the data displayed in the pane for an attribute by various properties.

In general, the pane for an attribute displays a table that shows incident data by the values of the attribute. The columns of the table include:

• Total calls made: Total number of API calls.
• % of calls: Percentage of all calls for each value of the attribute.
• Last detection time: The last time an event related to the incident was detected.

For some attributes, the table has additional columns.

You can select from the following attributes in the left-hand pane:

• API products: View incident details by API product.
• App keys: View incident details by app key—also known as API key or consumer key—an identifier for the client.
• Countries/Regions: View incident details by the countries and regions where the events in the incident originated.
• Developers: View incident details by developers—the people who use your APIs to develop applications. In addition to the three columns described above, Developers also has a column labeled Apps, which gives the number of applications for each developer.
• Developer apps: View incident details by application.

  In addition to the three columns described above, Developer apps also has the column Developers, which are the people who created the apps.

• IP Addresses: View incident details by IP addresses that are sources of events in the incident. Click View All IP Addresses to view the IP addresses. Note: The IP Addresses pane displays unique IP addresses, even if more than one incident corresponds to the same IP address.
  Note: To download a CSV file containing the IP addresses of detected traffic, click Download CSV File.

  IP Addresses displays the following columns:

  • IP address: The IP address for the incident.
  • Location: Location of the IP address.
  • Detected traffic: Total number of requests from the IP address.
  • % of calls: Percentage of requests from the IP address out of all calls in the environment.
  • First event detected: The first time an event was detected in the incident.
  • Last event detected: The last time an event was detected in the incident.
• Proxies: View incident details by proxy.
• Response codes: View incident details by response code.
• Rules: View incident details by detection rules.
• User agents: View incident details by user agent—the software agent that made the API call.

What does it mean when an attribute has the value (not set)?

Occasionally, an attribute has the value (not set). There are several reasons why this might occur. For one, Apigee might not have enough information to determine the attribute's value—for example, the country of origin of an API call. Or, the attribute might not apply in a particular case. See What does an analytics entity value "(not set)" mean? for more information.

Detected traffic

The Detected Traffic view displays information about incidents whose Last event detected is within the past 14 days. See Limitations on incidents and data displayed for more information on the time range of the data displayed in the UI.

To open the Detected Traffic view, select Detected Traffic in the Environment details view, as shown below:

The Detected Traffic view displays data for:

• Total traffic: The total number of requests.
• Detected traffic: The number of requests from IP addresses of detected abuse.
• % of detected traffic: The percentage that detected traffic makes up of out of total traffic.
• Detected IP address count: The number of distinct IP addresses corresponding to the detected abuse. Multiple requests from the same IP address are only counted once.

The Detected Traffic view also displays a table listing the details of each IP address corresponding to the detected abuse. Note that by default, the IP addresses are not displayed for privacy reasons. To view them, select Display all IP addresses at the top of the table.

Each row of the IP address table displays:

• IP address: IP address of the detected abuse. Click View to see the address.
• Location: The location of the IP address.
• Top app key: The app key used most frequently in requests from the IP address. Note: app key is another term for API key.
• Detection rules: A list of all detection rules that were triggered by the abuse.
• Top URL: The URL that received the most requests from the IP address.
• Detected traffic: The number of requests from the IP address.
• % of detected traffic: The percentage of requests from the IP address out of all requests in the environment.
• First event detected: The first time an event was detected in a request from the IP address during the time range selected at the top of the Security Scores page.
• Last event detected: The last time an event was detected in a request from the IP address during the time range selected at the top of the Security Scores page.

Limitations on abuse detection

Abuse detection has the following limitations.

• Incidents whose Last event detected was over 14 days in the past are not displayed in the the Abuse detection UI. See Limitations on incidents and data displayed for more information on what incidents and data are displayed in the UI.
• When you first enable Advanced API for an organization, or re-enable it, there will be a delay while events are clustered into incidents. After that, there will be periodic delays.
• The Incident details attribute page can take a brief time to load for organizations with a large amount of traffic.