This page describes troubleshooting methods for errors you may encounter while using App Engine.
See the Google Cloud Service Health Dashboard for information about incidents affecting Google Cloud services such as App Engine.
Deployment
If you run into problems deploying your app using the App Engine Admin API, this section lists error messages and suggestions for fixing each error.
The caller does not have permission to access project
The following error occurs when you deploy your app:
User EMAIL_ADDRESS does not have permission to access project PROJECT_ID (or it may not exist): The caller does not have permission
This error occurs if the account that you used to deploy your app does not have permission to deploy apps for the current project.
To resolve this issue,
grant the
App Engine Deployer (roles/appengine.deployer
) role
to the account. To see which account you used to deploy, do one of the
following:
- If you used the Google Cloud CLI to deploy, run the
gcloud auth list
command. - If you deployed from an IDE, view the settings for the Cloud Tools plugin.
Failed to fetch metadata from the registry
The following error occurs when you deploy your app:
Failed to fetch metadata from the registry, with reason: generic::permission_denied
This error occurs if you use the gcloud app deploy
command from a service
account that does not have the Storage Admin (roles/storage.admin
) role.
To resolve this issue, grant the Storage Admin role to the service account:
- To see which account you used, run the
gcloud auth list
command. - To learn why assigning only the App Engine Deployer
(
roles/appengine.deployer
) role might not be sufficient in some cases, see App Engine roles.
Service accounts must have permissions on the image
The following error occurs when you deploy your app:
The App Engine appspot and App Engine flexible environment service accounts must have permissions on the image IMAGE_NAME
This error occurs for one of the following reasons:
The default App Engine service account does not have the Storage Object Viewer (
roles/storage.objectViewer
) role.- To resolve this issue, grant the Storage Object Viewer role to the service account.
Your project has a VPC Service Perimeter which limits access to the Cloud Storage API using access levels.
- To resolve this issue, add the service account you use to deploy your app to the corresponding VPC Service Perimeter accessPolicies.
Failed to create Cloud Build
The following error occurs when you deploy your app:
Failed to create cloud build: Permission denied
This error occurs if you use the gcloud app deploy
command from an account that
does not have the Cloud Build Editor (roles/cloudbuild.builds.editor
) role.
To resolve this issue, grant the Cloud Build Editor role to the service account that you are using to deploy your app.
To see which account you used, run the
gcloud auth list
command.
Permissions error fetching application
The following error occurs when you deploy your app:
Permissions error fetching application apps/app_name. Please make sure you are using the correct project ID and that you have permission to view applications on the project
If you are running Google Cloud CLI version 328 or later, the following error occurs when you deploy your app:
Permissions error fetching application apps/app_name. Please make sure that you have permission to view applications on the project and that SERVICE_ACCOUNT has the App Engine Deployer (roles/appengine.deployer) role.
This error occurs if the account that you used to deploy your
app doesn't have the App Engine Deployer (roles/appengine.deployer
)
role.
To resolve this issue, verify that you have granted the App Engine Deployer role to the service account that you used to deploy your app; grant the role if the service account does not have it. To see which account you used to deploy, do one of the following:
- If you used the Google Cloud CLI to deploy, run the
gcloud auth list
command. - If you deployed from an IDE, view the settings for the Cloud Tools plugin.
Deployment fails for new projects
You might see the following error when you deploy your app for the first time in a new project:
ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: com.google.net.rpc3.client.RpcClientException:..........invalid bucket "staging.PROJECT-ID.appspot.com"; service account PROJECT-ID@appspot.gserviceaccount.com does not have access to the bucket
Various factors can cause this error, such as missing permissions, and changes to org policies.
To resolve this issue, verify that you have granted the Storage Admin (roles/storage.admin
) role to the
default service account. To grant the Storage Admin role, see Store build logs in a user-created bucket
If you have already granted the Storage Admin role, along with the other required roles based on different permission errors you encounter during deployment, and are still unable to deploy your app, it might be due to the following changes to org policies:
Starting in May 2024, Google Cloud enforces secure-by-default organization policies for all organization resources. This policy prevents App Engine from granting the
Editor
role to the App Engine default services accounts by default.Starting in June 2024, Cloud Build has changed the default behavior for how Cloud Build uses service accounts in new projects. This change is detailed in Cloud Build Service Account Change. As a result of this change, new projects deploying versions for the first time may be using the default App Engine service account with insufficient permissions for deploying versions.
To resolve this issue, do the following:
Grant the
Editor
role to the App Engine default service account (PROJECT_ID@appspot.gserviceaccount.com
).Review the Cloud Build guidance on changes to the default service account and opt out of the default changes in new projects.
Timed out waiting for the app infrastructure to become healthy
The following error occurs when you deploy your app:
Timed out waiting for the app infrastructure to become healthy
Various factors can cause this error, such as missing permissions, code errors, insufficient CPU or memory, or failed health checks. The error only occurs in the App Engine flexible environment.
To resolve this issue, rule out the following potential causes:
- Verify that you have granted the
Editor (
roles/editor
) role to your default App Engine service account. - Check whether the organization policy for your project restricts access to external IP addresses. For more information, see App Engine flexible environment known issues.
Verify that you have granted the following roles to the service account that you use to run your application (usually the default service account,
app-id@appspot.gserviceaccount.com
):Grant the roles if the service account does not have them.
If you are deploying in Shared VPC setup and passing
instance_tag
inapp.yaml
, refer to this section to fix the issue.
Permissions error when deploying a service with Serverless VPC Access Connector
The following error occurs when you deploy your app:
Please ensure you have [compute.globalOperations.get] on the service project
This error occurs when the user or service account that is trying to deploy the app with Serverless VPC Connector does not have the required permissions.
To resolve this issue, ensure the user or service account doing the deployment has Serverless VPC Access User and Compute Viewer IAM roles.
Invalid value error when deploying in a Shared VPC setup
The following error shows in Cloud Logging for Flex VM instances when you deploy your app:
Invalid value for field 'resource.tags.items[1]': 'aef-instance'. Duplicate tags are not allowed: aef-instance on compute.instances.insert
This error is because of a current known issue where setting the instance_tag results in errors when creating instances.
To resolve the issue, remove the instance_tag
field from your app.yaml
file and redeploy.
Errors when deploying apps with three or less maximum instances
Apps deployed with max_instances
set to three or less might experience unexpected errors or downtime. To resolve the
issue, specify at least four maximum instances in your app.yaml
file and redeploy.
Error when exceeding the limit on maximum instances
The following error occurs when you deploy your app:
You may not have more than 'xx' total max instances in your project.
There is a limit for maximum number of instances you can create per project. Requests to create additional instances fail if you exceed this limit.
To resolve this issue, set the value of max_instances
in your app.yaml
file to a value less than this limit
or delete some services or versions to bring the sum of max_instances
within
the limit.
Error when restarting instances under running versions
The following error occurs when you deploy your app:
error when restarting the instance under the running versions
In May 2024, Google Cloud began enforcing the Secure-by-default organization policies for all new organizations. This policy requires all VM instances created in new projects to have VM Manager enabled. For new and existing projects, this constraint prevents metadata updates that disable VM Manager at the project or instance level.
To resolve this issue, you must disable the organization policy constraint
Require OS Config (constraints/compute.requireOsConfig
).
If this issue continues to persist, you must also disable the following org policies that might have been enabled at the project or the organization level:
Define allowed external IPs for VM instances (
constraints/compute.vmExternalIpAccess
). If your application is setup to only use private networking, you don't have to disable this constraint.Disable Guest Attributes of Compute Engine metadata (
constraints/compute.disableGuestAttributesAccess
)
Required 'compute.firewalls.list' permission
The following error occurs when you deploy your app on a Shared VPC network:
Request to https://compute.googleapis.com/compute/v1/projects/projects/PROJECT_ID/global/firewalls?key failed, details: Required 'compute.firewalls.list' permission for 'projects/PROJECT_ID'
This error occurs if the following service accounts for the
host project
don't have the Compute Network User (roles/compute.networkUser
) role:
To resolve this issue, verify that you have granted the Compute Network User role to the Google APIs Service Agent and App Engine flexible environment Service Agent service accounts for the host project; grant the role if the service accounts do not have it.
Build during deployment fails without errors in logs
The following error occurs when you are deploying your app:
ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com/cloud-build/builds/BUILD_ID?project=PROJECT_NUMBER Failure status: UNKNOWN: Error Response: [2] Build failed; check build logs for details
Following the link in the error message shows that all build steps were successful. However, the app failed to build.
This problem occurs if you are using customer-managed encryption keys
(CMEK) or you have set up a data retention policy
for your staging.PROJECT_ID.appspot.com
bucket.
To resolve this issue, change the following settings for your
staging.PROJECT_ID.appspot.com
bucket:
- Set encryption to Google-owned and Google-managed keys.
- Remove the retention policy.
Deployment fails due to an organizational policy constraint
The following error occurs when you deploy an app:
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred while processing task /app-engine-flex/....: Request to https://compute.googleapis.com/compute/[VERSSION]/[PROJECT_ID]/... failed, details: Constraint constraints/compute.disableGuestAttributesAccess violated for project [PROJECT_ID].
This error can occur due to the enforcement of the organization policy constraint constraints/compute.disableGuestAttributesAccess
when deploying your
app. This policy is enforced by default for all apps in the App Engine flexible environment.
To resolve this issue, you must disable the organization policy constraint
constraints/compute.disableGuestAttributesAccess
.
Errors when deploying to an existing flexible environment version
The following error might occur when deploying to an existing App Engine flexible version:
ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy
This error indicates that updating an unhealthy deployment with a working Docker image doesn't always result in a healthy deployment. The outcome depends on the state of instances from the unhealthy deployment. Despite the error, if you provide a good Docker image, the deployment might eventually become healthy. Updating an existing version with a new Docker image, though allowed, is not a good practice. There is no rollback in case of version failure.
Serving errors
The following sections cover app initialization and serving errors.
Permission error when creating an app with the default service account
When you create an app after enabling the App Engine API for the first time, it fails with the following errors:
gcloud CLI
An internal error occurred while calling service consumer manager for service account.
Creating App Engine application in project [PROJECT__ID] and [REGION]....failed. DEBUG: (gcloud.app.create) Error Response: [13] an internal error has occurred
Request logs
Service account creation is not allowed on this project.
Console
Error while initialising App Engine.
This error can occur due to the enforcement of the organization policy constraint constraints/iam.disableServiceAccountCreation
when creating your
app. This policy prevents the provisioning of the App Engine default service account PROJECT_ID@appspot.gserviceaccount.com
.
To resolve this issue, you must temporarily remove the organization policy
constraint constraints/iam.disableServiceAccountCreation
to allow for the
creation and deployment of the App Engine default service account. The default
service account is necessary for app creation and can't be skipped. This is also
applicable when you use a per-version service account.
The App Engine default service account can then be deleted or replaced
with a service account that you create after successful deployment.
If you are using a service account that you created, review the Overview of role recommendations to understand how to enforce restricting permissions, such as providing a token creator role on the service account you create for the service agent.
Nginx fails to connect or contact the app container
The following error only occurs in the App Engine flexible environment and typically returns with 502 errors immediately after the error:
recv() failed (104: Connection reset by peer) while reading response header from upstream
This error indicates that nginx reverse proxy (nginx sidecar) is unable to reach the app container. In the logs, you can compare the close timing of the 502 error in the nginx log with the timing of the nginx.error log. A nginx.error followed immediately by a 502 nginx error is likely the cause of the nginx 502 error.
This error often occurs when the connection keepalive timeout of the
application is smaller than the keepalive timeout of nginx. As nginx in the App Engine flexible environment
has keepalive_timeout
of 650 seconds, applications need to keep connections alive for
at least this long. By default, Node.js applications have
keepAliveTimeout
of
5000 milliseconds. In this case, you can set server.keepAliveTimeout
to 700000 milliseconds.
To troubleshoot, check the logs written by the code running in your app container by connecting to the VM instance, and add more logging, if necessary, to find the root cause.
Insufficient memory
The following error occurs in the App Engine flexible environment and typically returns with 502 errors:
kernel: [ 133.706951] Out of memory: Kill process 4490 (java) score 878 or sacrifice child kernel: [ 133.714468] Killed process 4306 (java) total-vm:5332376kB, anon-rss:2712108kB, file-rss:0kB
This error indicates that App Engine has terminated the application.
This error occurs when the instance has insufficient memory. By default App Engine flexible environment has 1GB of memory, with only 600MB available for the application container.
To troubleshoot, check the logs and if you see an Out of Memory entry. Update
your memory_gb
configuration and redeploy.
Insufficient open connections to handle incoming requests
Apps might encounter a 502 error if the maximum number of waiting connections is equal to or greater than 75% of the number of active connections.
To resolve the issue, check the Cloud Monitoring metrics for the maximum number of active and waiting connections, and decrease the number of waiting connections to ensure that the maximum number of waiting connections is less than or equal to 75% of the number of active connections.