Troubleshoot App Engine errors

This page describes troubleshooting methods for errors you may encounter while using App Engine.

See the Google Cloud Service Health Dashboard for information about incidents affecting Google Cloud services such as App Engine.

Deployment

If you run into problems deploying your app using the App Engine Admin API, this section lists error messages and suggestions for fixing each error.

The caller does not have permission to access project

The following error occurs when you deploy your app:

User EMAIL_ADDRESS does not have permission to access project PROJECT_ID (or it may not exist): The caller does not have permission

This error occurs if the account that you used to deploy your app does not have permission to deploy apps for the current project.

To resolve this issue, grant the App Engine Deployer (roles/appengine.deployer) role to the account. To see which account you used to deploy, do one of the following:

  • If you used the Google Cloud CLI to deploy, run the gcloud auth list command.
  • If you deployed from an IDE, view the settings for the Cloud Tools plugin.

Failed to fetch metadata from the registry

The following error occurs when you deploy your app:

Failed to fetch metadata from the registry, with reason: generic::permission_denied

This error occurs if you use the gcloud app deploy command from a service account that does not have the Storage Admin (roles/storage.admin) role.

To resolve this issue, grant the Storage Admin role to the service account:

  • To see which account you used, run the gcloud auth list command.
  • To learn why assigning only the App Engine Deployer (roles/appengine.deployer) role might not be sufficient in some cases, see App Engine roles.

Service accounts must have permissions on the image

The following error occurs when you deploy your app:

The App Engine appspot and App Engine flexible environment service accounts must
have permissions on the image IMAGE_NAME

This error occurs for one of the following reasons:

Failed to create Cloud Build

The following error occurs when you deploy your app:

Failed to create cloud build: Permission denied

This error occurs if you use the gcloud app deploycommand from an account that does not have the Cloud Build Editor (roles/cloudbuild.builds.editor) role.

To resolve this issue, grant the Cloud Build Editor role to the service account that you are using to deploy your app.

To see which account you used, run the gcloud auth list command.

Permissions error fetching application

The following error occurs when you deploy your app:

Permissions error fetching application apps/app_name. Please make sure you are using the correct project ID and that you have permission to view applications on the project

If you are running Google Cloud CLI version 328 or later, the following error occurs when you deploy your app:

Permissions error fetching application apps/app_name. Please
make sure that you have permission to view applications on the project and that
SERVICE_ACCOUNT has the App Engine Deployer (roles/appengine.deployer) role.

This error occurs if the account that you used to deploy your app doesn't have the App Engine Deployer (roles/appengine.deployer) role.

To resolve this issue, verify that you have granted the App Engine Deployer role to the service account that you used to deploy your app; grant the role if the service account does not have it. To see which account you used to deploy, do one of the following:

  • If you used the Google Cloud CLI to deploy, run the gcloud auth list command.
  • If you deployed from an IDE, view the settings for the Cloud Tools plugin.

Deployment fails for new projects

You might see the following error when you deploy your app for the first time in a new project:

ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: com.google.net.rpc3.client.RpcClientException:..........invalid bucket "staging.PROJECT-ID.appspot.com"; service account PROJECT-ID@appspot.gserviceaccount.com does not have access to the bucket

Various factors can cause this error, such as missing permissions, and changes to org policies.

To resolve this issue, verify that you have granted the Storage Admin (roles/storage.admin) role to the default service account. To grant the Storage Admin role, see Store build logs in a user-created bucket

If you have already granted the Storage Admin role, along with the other required roles based on different permission errors you encounter during deployment, and are still unable to deploy your app, it might be due to the following changes to org policies:

  • Starting in May 2024, Google Cloud enforces secure-by-default organization policies for all organization resources. This policy prevents App Engine from granting the Editor role to the App Engine default services accounts by default.

  • Starting in June 2024, Cloud Build has changed the default behavior for how Cloud Build uses service accounts in new projects. This change is detailed in Cloud Build Service Account Change. As a result of this change, new projects deploying versions for the first time may be using the default App Engine service account with insufficient permissions for deploying versions.

To resolve this issue, do the following:

Timed out waiting for the app infrastructure to become healthy

The following error occurs when you deploy your app:

Timed out waiting for the app infrastructure to become healthy

Various factors can cause this error, such as missing permissions, code errors, insufficient CPU or memory, or failed health checks. The error only occurs in the App Engine flexible environment.

To resolve this issue, rule out the following potential causes:

  1. Verify that you have granted the Editor (roles/editor) role to your default App Engine service account.
  2. Check whether the organization policy for your project restricts access to external IP addresses. For more information, see App Engine flexible environment known issues.

  3. Verify that you have granted the following roles to the service account that you use to run your application (usually the default service account, app-id@appspot.gserviceaccount.com):

  4. Grant the roles if the service account does not have them.

  5. If you are deploying in Shared VPC setup and passing instance_tag in app.yaml, refer to this section to fix the issue.

Permissions error when deploying a service with Serverless VPC Access Connector

The following error occurs when you deploy your app:

Please ensure you have [compute.globalOperations.get] on the service project

This error occurs when the user or service account that is trying to deploy the app with Serverless VPC Connector does not have the required permissions.

To resolve this issue, ensure the user or service account doing the deployment has Serverless VPC Access User and Compute Viewer IAM roles.

Invalid value error when deploying in a Shared VPC setup

The following error shows in Cloud Logging for Flex VM instances when you deploy your app:

Invalid value for field 'resource.tags.items[1]': 'aef-instance'. Duplicate
tags are not allowed: aef-instance on compute.instances.insert

This error is because of a current known issue where setting the instance_tag results in errors when creating instances.

To resolve the issue, remove the instance_tag field from your app.yaml file and redeploy.

Errors when deploying apps with three or less maximum instances

Apps deployed with max_instances set to three or less might experience unexpected errors or downtime. To resolve the issue, specify at least four maximum instances in your app.yaml file and redeploy.

Error when exceeding the limit on maximum instances

The following error occurs when you deploy your app:

You may not have more than 'xx' total max instances in your project.

There is a limit for maximum number of instances you can create per project. Requests to create additional instances fail if you exceed this limit.

To resolve this issue, set the value of max_instances in your app.yaml file to a value less than this limit or delete some services or versions to bring the sum of max_instances within the limit.

Error when restarting instances under running versions

The following error occurs when you deploy your app:

error when restarting the instance under the running versions

In May 2024, Google Cloud began enforcing the Secure-by-default organization policies for all new organizations. This policy requires all VM instances created in new projects to have VM Manager enabled. For new and existing projects, this constraint prevents metadata updates that disable VM Manager at the project or instance level.

To resolve this issue, you must disable the organization policy constraint Require OS Config (constraints/compute.requireOsConfig).

If this issue continues to persist, you must also disable the following org policies that might have been enabled at the project or the organization level:

Required 'compute.firewalls.list' permission

The following error occurs when you deploy your app on a Shared VPC network:

Request to https://compute.googleapis.com/compute/v1/projects/projects/PROJECT_ID/global/firewalls?key failed, details: Required 'compute.firewalls.list' permission for 'projects/PROJECT_ID'

This error occurs if the following service accounts for the host project don't have the Compute Network User (roles/compute.networkUser) role:

To resolve this issue, verify that you have granted the Compute Network User role to the Google APIs Service Agent and App Engine flexible environment Service Agent service accounts for the host project; grant the role if the service accounts do not have it.

Build during deployment fails without errors in logs

The following error occurs when you are deploying your app:

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com/cloud-build/builds/BUILD_ID?project=PROJECT_NUMBER Failure status: UNKNOWN: Error Response: [2] Build failed; check build logs for details

Following the link in the error message shows that all build steps were successful. However, the app failed to build.

This problem occurs if you are using customer-managed encryption keys (CMEK) or you have set up a data retention policy for your staging.PROJECT_ID.appspot.com bucket.

To resolve this issue, change the following settings for your staging.PROJECT_ID.appspot.com bucket:

Deployment fails due to an organizational policy constraint

The following error occurs when you deploy an app:

ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred while processing task /app-engine-flex/....: Request to https://compute.googleapis.com/compute/[VERSSION]/[PROJECT_ID]/... failed, details: Constraint constraints/compute.disableGuestAttributesAccess violated for project [PROJECT_ID].

This error can occur due to the enforcement of the organization policy constraint constraints/compute.disableGuestAttributesAccess when deploying your app. This policy is enforced by default for all apps in the App Engine flexible environment.

To resolve this issue, you must disable the organization policy constraint constraints/compute.disableGuestAttributesAccess.

Errors when deploying to an existing flexible environment version

The following error might occur when deploying to an existing App Engine flexible version:

ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while
processing task /app-engine-flex/flex_await_healthy/flex_await_healthy

This error indicates that updating an unhealthy deployment with a working Docker image doesn't always result in a healthy deployment. The outcome depends on the state of instances from the unhealthy deployment. Despite the error, if you provide a good Docker image, the deployment might eventually become healthy. Updating an existing version with a new Docker image, though allowed, is not a good practice. There is no rollback in case of version failure.

Serving errors

The following sections cover app initialization and serving errors.

Permission error when creating an app with the default service account

When you create an app after enabling the App Engine API for the first time, it fails with the following errors:

gcloud CLI 

An internal error occurred while calling service consumer manager for service account.
Creating  App Engine application in project [PROJECT__ID] and [REGION]....failed. DEBUG: (gcloud.app.create) Error Response: [13] an internal error has occurred

Request logs 

Service account creation is not allowed on this project.

Console 

Error while initialising App Engine.

This error can occur due to the enforcement of the organization policy constraint constraints/iam.disableServiceAccountCreation when creating your app. This policy prevents the provisioning of the App Engine default service account PROJECT_ID@appspot.gserviceaccount.com.

To resolve this issue, you must temporarily remove the organization policy constraint constraints/iam.disableServiceAccountCreation to allow for the creation and deployment of the App Engine default service account. The default service account is necessary for app creation and can't be skipped. This is also applicable when you use a per-version service account. The App Engine default service account can then be deleted or replaced with a service account that you create after successful deployment.

If you are using a service account that you created, review the Overview of role recommendations to understand how to enforce restricting permissions, such as providing a token creator role on the service account you create for the service agent.

Nginx fails to connect or contact the app container

The following error only occurs in the App Engine flexible environment and typically returns with 502 errors immediately after the error:

recv() failed (104: Connection reset by peer) while reading response header from upstream

This error indicates that nginx reverse proxy (nginx sidecar) is unable to reach the app container. In the logs, you can compare the close timing of the 502 error in the nginx log with the timing of the nginx.error log. A nginx.error followed immediately by a 502 nginx error is likely the cause of the nginx 502 error.

This error often occurs when the connection keepalive timeout of the application is smaller than the keepalive timeout of nginx. As nginx in the App Engine flexible environment has keepalive_timeout of 650 seconds, applications need to keep connections alive for at least this long. By default, Node.js applications have keepAliveTimeout of 5000 milliseconds. In this case, you can set server.keepAliveTimeout to 700000 milliseconds.

To troubleshoot, check the logs written by the code running in your app container by connecting to the VM instance, and add more logging, if necessary, to find the root cause.

Insufficient memory

The following error occurs in the App Engine flexible environment and typically returns with 502 errors:

kernel: [  133.706951] Out of memory: Kill process 4490 (java) score 878 or sacrifice child
kernel: [  133.714468] Killed process 4306 (java) total-vm:5332376kB, anon-rss:2712108kB, file-rss:0kB

This error indicates that App Engine has terminated the application.

This error occurs when the instance has insufficient memory. By default App Engine flexible environment has 1GB of memory, with only 600MB available for the application container.

To troubleshoot, check the logs and if you see an Out of Memory entry. Update your memory_gb configuration and redeploy.

Insufficient open connections to handle incoming requests

Apps might encounter a 502 error if the maximum number of waiting connections is equal to or greater than 75% of the number of active connections.

To resolve the issue, check the Cloud Monitoring metrics for the maximum number of active and waiting connections, and decrease the number of waiting connections to ensure that the maximum number of waiting connections is less than or equal to 75% of the number of active connections.