Problem
You configure a Cloud Monitoring alert policy to create an incident for 5xx HTTP responses served by an App Engine app. However, when the policy creates an incident, there are no requests with status 5xx in the Google App Engine application log.
Environment
- App Engine Flexible Environment
- Cloud Monitoring Alert policy
- Resource: Google App Engine Application
- Metric: Response count
- Filter: response_code >= 500
- Cloud Monitoring log investigated:
resource.type="gae_app" logName=( "projects/REDACTED/logs/appengine.googleapis.com%2Fstdout" OR "projects/REDACTED/logs/appengine.googleapis.com%2Fstderr" OR "projects/REDACTED/logs/appengine.googleapis.com%2Fnginx.request" )
Solution
The 5xx responses can be seen in the HTTP Load Balancer logs:
resource.type="http_load_balancer" httpRequest.status=502
Cause
It is possible that 5xx errors are monitored but not visible in the App Engine logs because the logs are produced on the instances while the monitoring (for App Engine Flexible Environment) is taken from the HTTP Load Balancer.
Especially 502 responses can happen if the HTTP Load Balancer can not find an appropriate instance to forward requests.