Monitoring and alerting latency

This page describes how to monitor and get alerts from elevated latency in Google App Engine.

High latency for App Engine apps can occur for various reasons. For example, the app does not have optimized code, there aren't enough resources, there are busy machines, and so forth. This guide focuses on the latency generated by App Engine regardless of the reason.

It is important to understand that high latency can often lead to 5xx errors. Therefore, rapid detection of high latency can help you act in time and thus resolve application errors.

Setting up Monitoring and Alerting

Monitoring

If you want to create a customized graph to monitor your app's response latency, use Cloud Monitoring > Metrics Explorer to build your own graph.

Go to Metrics Explorer page

In the Resource Type, select GAE Application and select Response latency in the Metric filter. If you have a specific service to check, you can also use Filters to filter by the specific module to see the response latency chart.

Screenshot of metric explorer

Now that you've set up your customized graph, let's take a look at the next section and learn how to set up notifications for latencies over a certain amount of seconds.

Alerting

To create an alerting policy, do the following:

  1. Set up notifications, create an alerting policy in Cloud Monitoring > Alerting and click on + CREATE POLICY.
  2. Add a condition to your alert by clicking ADD CONDITION.

    Go to Alerting page

  3. Select GAE Application as a target:

For example, if you wanted to get alerts for response latency, such as how long your App Engine app is taking to respond, select response latency.

Screenshot of create an alerting policy

You can also select the module ID if it's appropriate, and add a threshold and a notification channel. See Managing notification channels for instructions.

After you have finished creating the policy, you will see the policy in a list in the Alerting page. Click on the created policy to see the details, as shown in the following image:

Screenshot of policy detail

Congratulations, you just created your first App Engine latency policy! After you create your policy, the app will send an alert to the selected channel if it detects an incident.

Debugging latency

Checking the App Engine latency dashboard

In general, when you want to see the latency of App Engine applications, go to App Engine > Dashboard and select the Latency graph.

Go to App Engine dashboard

In the latency graph, you can drill into the data by filtering the graphs by service, version, and interval.

Screenshot of gae dashboard latency graph

In the previous image, notice how the application latency varies from a few milliseconds to almost 25s.

Understanding app latency

To understand now how long a particular request has taken, go to:

Cloud Logging

In Cloud Logging you can find two categories of logs, request logs and application logs. Request logs are automatically written by App Engine for each request to your application and contains the HTTP method, response code, and so forth. Application logs are those that you write for your application.

To view request logs or application logs, see Viewing logs.

Screenshot of application and request logs in Logs Explorer

Cloud Trace

Cloud Trace is useful for understanding how requests propagate through your application. You can inspect detailed latency information for a single request or view aggregate latency for your entire application.

To view trace details in Cloud Trace, you can follow Find and explore traces. In the Trace explorer, as shown in the following image, you can use the filters to filter by your specific App Engine service and version.

Screenshot of trace explorer