Troubleshooting Cloud Functions

This document shows you some of the common problems you might run into and how to deal with them.

Deployment

The deployment phase is a frequent source of problems. Many of the issues you might encounter during deployment are related to roles and permissions. Others have to do with incorrect configuration.

User with Viewer role cannot deploy a function

A user who has been assigned the Project Viewer or Cloud Functions Viewer role has read-only access to functions and function details. These roles are not allowed to deploy new functions.

The error message

Cloud Console

You need permissions for this action. Required permission(s): cloudfunctions.functions.create

Cloud SDK

ERROR: (gcloud.functions.deploy) PERMISSION_DENIED: Permission
'cloudfunctions.functions.sourceCodeSet' denied on resource
'projects/<PROJECT_ID>/locations/<LOCATION>` (or resource may not exist)

The solution

Assign the user a role that has the appropriate access.

User with Project Viewer or Cloud Function role cannot deploy a function

In order to deploy a function, a user who has been assigned the Project Viewer, the Cloud Function Developer, or Cloud Function Admin role must be assigned an additional role.

The error message

Cloud Console

User does not have the iam.serviceAccounts.actAs permission on
<PROJECT_ID>@appspot.gserviceaccount.com required to create function.
You can fix this by running
'gcloud iam service-accounts add-iam-policy-binding <PROJECT_ID>@appspot.gserviceaccount.com --member=user: --role=roles/iam.serviceAccountUser'

Cloud SDK

ERROR: (gcloud.functions.deploy) ResponseError: status=[403], code=[Forbidden],
message=[Missing necessary permission iam.serviceAccounts.actAs for <USER>
on the service account <PROJECT_ID>@appspot.gserviceaccount.com. Ensure that
service account <PROJECT_ID>@appspot.gserviceaccount.com is a member of the
project <PROJECT_ID>, and then grant <USER> the role 'roles/iam.serviceAccountUser'.
You can do that by running
'gcloud iam service-accounts add-iam-policy-binding <PROJECT_ID>@appspot.gserviceaccount.com --member=<USER> --role=roles/iam.serviceAccountUser'
In case the member is a service account please use the prefix 'serviceAccount:' instead of 'user:'.]

The solution

Assign the user an additional role, the Service Account User IAM role (roles/iam.serviceAccountUser), scoped to the Cloud Functions runtime service account.

Deployment service account missing the Service Agent role when deploying functions

The Cloud Functions service uses the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) when performing administrative actions on your project. By default this account is assigned the Cloud Functions cloudfunctions.serviceAgent role. This role is required for Cloud Pub/Sub, IAM, Cloud Storage and Firebase integrations. If you have changed the role for this service account, deployment fails.

The error message

Cloud Console

Missing necessary permission resourcemanager.projects.getIamPolicy for
serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com on project <PROJECT_ID>.
Please grant serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
the roles/cloudfunctions.serviceAgent role. You can do that by running
'gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com --role=roles/cloudfunctions.serviceAgent'

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=7,
message=Missing necessary permission resourcemanager.projects.getIamPolicy
for serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
on project <PROJECT_ID>. Please grant
serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
the roles/cloudfunctions.serviceAgent role. You can do that by running
'gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com --role=roles/cloudfunctions.serviceAgent'

The solution

Reset this service account to the default role.

Deployment service account missing Pub/Sub permissions when deploying an event-driven function

The Cloud Functions service uses the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) when performing administrative actions. By default this account is assigned the Cloud Functions cloudfunctions.serviceAgent role. To deploy event-driven functions, the Cloud Functions service must access Cloud Pub/Sub to configure topics and subscriptions. If the role assigned to the service account is changed and the appropriate permissions are not otherwise granted, the Cloud Functions service cannot access Cloud Pub/Sub and the deployment fails.

The error message

Cloud Console

Failed to configure trigger PubSub projects/<PROJECT_ID>/topics/<FUNCTION_NAME>

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=13,
message=Failed to configure trigger PubSub projects/<PROJECT_ID>/topics/<FUNCTION_NAME>

The solution

You can:

  • Reset this service account to the default role.

    or

  • Grant the pubsub.subscriptions.* and pubsub.topics.* permissions to your service account manually.

User missing permissions for runtime service account while deploying a function

In environments where multiple functions are accessing different resources, it is a common practice to use per-function identities, with named runtime service accounts rather than the default runtime service account (PROJECT_ID@appspot.gserviceaccount.com).

However, to use a non-default runtime service account, the deployer must have the iam.serviceAccounts.actAs permission on that non-default account. A user who creates a non-default runtime service account is automatically granted this permission, but other deployers must have this permission granted by a user with the correct permissions.

The error message

Cloud SDK

ERROR: (gcloud.functions.deploy) ResponseError: status=[400], code=[Bad Request],
message=[Invalid function service account requested: <SERVICE_ACCOUNT_NAME@<PROJECT_ID>.iam.gserviceaccount.com]

The solution

Assign the user the roles/iam.serviceAccountUser role on the non-default <SERVICE_ACCOUNT_NAME> runtime service account. This role includes the iam.serviceAccounts.actAs permission.

Runtime service account missing project bucket permissions while deploying a function

Cloud Functions can only be triggered by events from Cloud Storage buckets in the same Google Cloud Platform project. In addition, the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) needs a cloudfunctions.serviceAgent role on your project.

The error message

Cloud Console

Deployment failure: Insufficient permissions to (re)configure a trigger
(permission denied for bucket <BUCKET_ID>). Please, give owner permissions
to the editor role of the bucket and try again.

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=7, message=Insufficient
permissions to (re)configure a trigger (permission denied for bucket <BUCKET_ID>).
Please, give owner permissions to the editor role of the bucket and try again.

The solution

You can:

  • Reset this service account to the default role.

    or

  • Grant the runtime service account the cloudfunctions.serviceAgent role.

    or

  • Grant the runtime service account the storage.buckets.{get, update} and the resourcemanager.projects.get permissions.

User with Project Editor role cannot make a function public

To ensure that unauthorized developers cannot modify authentication settings for function invocations, the user or service that is deploying the function must have the cloudfunctions.functions.setIamPolicy permission.

The error message

Cloud SDK

ERROR: (gcloud.functions.add-iam-policy-binding) ResponseError: status=[403], code=[Forbidden], message=[Permission 'cloudfunctions.functions.setIamPolicy' denied on resource 'projects/<PROJECT_ID>/locations/<LOCATION>/functions/<FUNCTION_NAME> (or resource may not exist).]

The solution

You can:

Function deployment fails due to Cloud Build not supporting VPC-SC

Cloud Functions uses Cloud Build to build your source code into a runnable container. In order to use Cloud Functions with VPC Service Controls, you must configure an access level for the Cloud Build service account in your service perimeter.

The error message

Cloud Console

One of the below:

Error in the build environment

OR

Unable to build your function due to VPC Service Controls. The Cloud Build
service account associated with this function needs an appropriate access
level on the service perimeter. Please grant access to the Cloud Build
service account: '{PROJECT_NUMBER}@cloudbuild.gserviceaccount.com' by
following the instructions at
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls#grant-build-access"

Cloud SDK

One of the below:

ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Error in
the build environment

OR

Unable to build your function due to VPC Service Controls. The Cloud Build
service account associated with this function needs an appropriate access
level on the service perimeter. Please grant access to the Cloud Build
service account: '{PROJECT_NUMBER}@cloudbuild.gserviceaccount.com' by
following the instructions at
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls#grant-build-access"

The solution

If your project's Audited Resources logs mention "Request is prohibited by organization's policy" in the VPC Service Controls section and have a Cloud Storage label, you need to grant the Cloud Build Service Account access to the VPC Service Controls perimeter.

Function deployment fails due to incorrectly specified entry point

Cloud Functions deployment can fail if the entry point to your code, that is, the exported function name, is not specified correctly.

The error message

Cloud Console

Deployment failure: Function failed on loading user code. Error message:
Error: please examine your function logs to see the error cause:
https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function
failed on loading user code. Error message: Please examine your function
logs to see the error cause:
https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs

The solution

Your source code must contain an entry point function that has been correctly specified in your deployment, either via Cloud Console or Cloud SDK.

Function deployment fails when using Resource Location Constraint organization policy

If your organization uses a Resource Location Constraint policy, you may see this error in your logs. It indicates that the deployment pipeline failed to create a multi-regional storage bucket.

The error message

In Cloud Build logs:

Token exchange failed for project '<PROJECT_ID>'.
Org Policy Violated: '<REGION>' violates constraint 'constraints/gcp.resourceLocations'

In Cloud Storage logs:

<REGION>.artifacts.<PROJECT_ID>.appspot.com` storage bucket could not be created.

The solution

If you are using constraints/gcp.resourceLocations in your organization policy constraints, you should specify the appropriate multi-region location. For example, if you are deploying in any of the us regions, you should use us-locations.

However, if you require more fine grained control and want to restrict function deployment to a single region (not multiple regions), create the multi-region bucket first:

  1. Allow the whole multi-region
  2. Deploy a test function
  3. After the deployment has succeeded, change the organizational policy back to allow only the specific region.

The multi-region storage bucket stays available for that region, so that subsequent deployments can succeed. If you later decide to allowlist a region outside of the one where the multi-region storage bucket was created, you must repeat the process.

Function deployment fails while executing function's global scope

This error indicates that there was a problem with your code. The deployment pipeline finished deploying the function, but failed at the last step - sending a health check to the function. This health check is meant to execute a function's global scope, which could be throwing an exception, crashing, or timing out. The global scope is where you commonly load in libraries and initialize clients.

The error message

In Cloud Logging logs:

"Function failed on loading user code. This is likely
due to a bug in the user code."

The solution

For a more detailed error message, look into your function's build logs, as well as your function's runtime logs. If it is unclear why your function failed to execute its global scope, consider temporarily moving the code into the request invocation, using lazy initialization of the global variables. This allows you to add extra log statements around your client libraries, which could be timing out on their instantiation (especially if they are calling other services), or crashing/throwing exceptions altogether.

Build

When you deploy your function's source code to Cloud Functions, that source is stored in a Cloud Storage bucket. Cloud Build then automatically builds your code into a container image and pushes that image to Container Registry. Cloud Functions accesses this image when it needs to run the container to execute your function.

Build failed due to missing Container Registry Images

Cloud Functions uses Container Registry to manage images of the functions. Container Registry uses Cloud Storage to store the layers of the images in buckets named STORAGE-REGION.artifacts.PROJECT-ID.appspot.com. Using Object Lifecycle Management on these buckets breaks the deployment of the functions as the deployments depend on these images being present.

The error message

Cloud Console

Build failed: Build error details not available. Please check the logs at
<CLOUD_CONSOLE_LINK>

CLOUD_CONSOLE_LINK contains an error like below :
failed to get OS from config file for image 'us.gcr.io/<PROJECT_ID>/gcf/us-central1/<UUID>/worker:latest'"

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Build
failed: Build error details not available. Please check the logs at
<CLOUD_CONSOLE_LINK>

CLOUD_CONSOLE_LINK contains an error like below :
failed to get OS from config file for image 'us.gcr.io/<PROJECT_ID>/gcf/us-central1/<UUID>/worker:latest'"

The solution

  1. Disable Lifecycle Management on the buckets required by Container Registry.
  2. Delete all the images of affected functions. You can access build logs to find the image paths. Reference script to bulk delete the images. Note that this does not affect the functions that are currently deployed.
  3. Redeploy the functions.

Serving

The serving phase can also be a source of errors.

Serving permission error due to the function being private

Cloud Functions allows you to declare functions private, that is, to restrict access to end users and service accounts with the appropriate permission. By default deployed functions are set as private. This error message indicates that the caller does not have permission to invoke the function.

The error message

HTTP Error Response code: 403 Forbidden

HTTP Error Response body: Error: Forbidden Your client does not have permission to get URL /<FUNCTION_NAME> from this server.

The solution

You can:

Serving permission error due to "allow internal traffic only" configuration

Ingress settings restrict whether an HTTP function can be invoked by resources outside of your Google Cloud project or VPC Service Controls service perimeter. When the "allow internal traffic only" setting for ingress networking is configured, this error message indicates that only requests from VPC networks in the same project or VPC Service Controls perimeter are allowed.

The error message

HTTP Error Response code: 403 Forbidden

HTTP Error Response body: Error 403 (Forbidden) 403. That's an error. Access is forbidden. That's all we know.

The solution

You can:

  • Ensure that the request is coming from your Google Cloud project or VPC Service Controls service perimeter.

    or

  • Change the ingress settings to allow all traffic for the function.

Function invocation lacks valid authentication credentials

Invoking a Cloud Functions function that has been set up with restricted access requires an ID token. Access tokens or refresh tokens do not work.

The error message

HTTP Error Response code: 401 Unauthorized

HTTP Error Response body: Your client does not have permission to the requested URL

The solution

Make sure that your requests include an Authorization: Bearer ID_TOKEN header, and that the token is an ID token, not an access or refresh token. If you are generating this token manually with a service account's private key, you must exchange the self-signed JWT token for a Google-signed Identity token, following this guide.

Attempt to invoke function using curl redirects to Google login page

If you attempt to invoke a function that does not exist, Cloud Functions responds with an HTTP/2 302 redirect which takes you to the Google account login page. This is incorrect. It should respond with an HTTP/2 404 error response code. The problem is being addressed.

The solution

Make sure you specify the name of your function correctly. You can always check using gcloud functions call which returns the correct 404 error for a missing function.

Application crashes and function execution fails

This error indicates that the process running your function has died. This is usually due to the runtime crashing due to issues in the function code. This may also happen when a deadlock or some other condition in your function's code causes the runtime to become unresponsive to incoming requests.

The error message

In Cloud Logging logs: "Infrastructure cannot communicate with function. There was likely a crash or deadlock in the user-provided code."

The solution

Different runtimes can crash under different scenarios. To find the root cause, output detailed debug level logs, check your application logic, and test for edge cases.

The Cloud Functions Python37 runtime currently has a known limitation on the rate that it can handle logging. If log statements from a Python37 runtime instance are written at a sufficiently high rate, it can produce this error. Python runtime versions >= 3.8 do not have this limitation. We encourage users to migrate to a higher version of the Python runtime to avoid this issue.

If you are still uncertain about the cause of the error, check out our support page.

Function stops mid-execution, or continues running after your code finishes

Some Cloud Functions runtimes allow users to run asynchronous tasks. If your function creates such tasks, it must also explicitly wait for these tasks to complete. Failure to do so may cause your function to stop executing at the wrong time.

The error behavior

Your function exhibits one of the following behaviors:

  • Your function terminates while asynchronous tasks are still running, but before the specified timeout period has elapsed.
  • Your function does not stop running when these tasks finish, and continues to run until the timeout period has elapsed.

The solution

If your function terminates early, you should make sure all your function's asynchronous tasks have been completed before doing any of the following:

  • returning a value
  • resolving or rejecting a returned Promise object (Node.js functions only)
  • throwing uncaught exceptions and/or errors
  • sending an HTTP response
  • calling a callback function

If your function fails to terminate once all asynchronous tasks have completed, you should verify that your function is correctly signaling Cloud Functions once it has completed. In particular, make sure that you perform one of the operations listed above as soon as your function has finished its asynchronous tasks.

JavaScript heap out of memory

For Node.js 12+ functions with memory limits greater than 2GiB, users need to configure NODE_OPTIONS to have max_old_space_size so that the JavaScript heap limit is equivalent to the function's memory limit.

The error message

Cloud Console

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The solution

Deploy your Node.js 12+ function, with NODE_OPTIONS configured to have max_old_space_size set to your function's memory limit. For example:

gcloud functions deploy envVarMemory \
--runtime nodejs14 \
--set-env-vars NODE_OPTIONS="--max_old_space_size=8192" \
--memory 8Gi \
--trigger-http

Function terminated

You may see one of the following error messages when the process running your code exited either due to a runtime error or a deliberate exit. There is also a small chance that a rare infrastructure error occurred.

The error messages

Function invocation was interrupted. Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting information can be found in Logging.

Request rejected. Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting information can be found in Logging.

Function cannot be initialized. Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting information can be found in Logging.

The solution

  • For a background (Pub/Sub triggered) function when an executionID is associated with the request that ended up in error, try enabling retry on failure. This allows the retrying of function execution when a retriable exception is raised. For more information for how to use this option safely, including mitigations for avoiding infinite retry loops and managing retriable/fatal errors differently, see Best Practices.

  • Background activity (anything that happens after your function has terminated) can cause issues, so check your code. Cloud Functions does not guarantee any actions other than those that run during the execution period of the function, so even if an activity runs in the background, it might be terminated by the cleanup process.

  • In cases when there is a sudden traffic spike, try spreading the workload over a little more time. Also test your functions locally using the Functions Framework before you deploy to Cloud Functions to ensure that the error is not due to missing or conflicting dependencies.

Scalability

Scaling issues related to Cloud Functions infrastructure can arise in several circumstances.

The following conditions can be associated with scaling failures.

  • A huge sudden increase in traffic.
  • A long cold start time.
  • A long request processing time.
  • High function error rate.
  • Reaching the maximum instance limit and hence the system cannot scale any further.
  • Transient factors attributed to the Cloud Functions service.

In each case Cloud Functions might not scale up fast enough to manage the traffic.

The error message

  • The request was aborted because there was no available instance
    • severity=WARNING ( Response code: 429 ) Cloud Functions cannot scale due to the max-instances limit you set during configuration.
    • severity=ERROR ( Response code: 500 ) Cloud Functions intrinsically cannot manage the rate of traffic.

The solution

  • For HTTP trigger-based functions, have the client implement exponential backoff and retries for requests that must not be dropped.
  • For background / event-driven functions, Cloud Functions supports at least once delivery. Even without explicitly enabling retry, the event is automatically re-delivered and the function execution will be retried. See Retrying Event-Driven Functions for more information.
  • When the root cause of the issue is a period of heightened transient errors attributed solely to Cloud Functions or if you need assistance with your issue, please contact support

Logging

Setting up logging to help you track down problems can cause problems of its own.

Logs entries have no, or incorrect, log severity levels

Cloud Functions includes simple runtime logging by default. Logs written to stdout or stderr appear automatically in the Cloud Console. But these log entries, by default, contain only simple string messages.

The error message

No or incorrect severity levels in logs.

The solution

To include log severities, you must send a structured log entry instead.

Handle or log exceptions differently in the event of a crash

You may want to customize how you manage and log crash information.

The solution

Wrap your function is a try/catch block to customize handling exceptions and logging stack traces.

Example


import logging
import traceback
def try_catch_log(wrapped_func):
  def wrapper(*args, **kwargs):
    try:
      response = wrapped_func(*args, **kwargs)
    except Exception:
      # Replace new lines with spaces so as to prevent several entries which
      # would trigger several errors.
      error_message = traceback.format_exc().replace('\n', '  ')
      logging.error(error_message)
      return 'Error';
    return response;
  return wrapper;


#Example hello world function
@try_catch_log
def python_hello_world(request):
  request_args = request.args

  if request_args and 'name' in request_args:
    1 + 's'
  return 'Hello World!'

Logs too large in Node.js 10+, Python 3.8, Go 1.13, and Java 11

The max size for a regular log entry in these runtimes is 105 KiB.

The solution

Make sure you send log entries smaller than this limit.

Cloud Functions logs are not appearing via Log Router Sink

Log entries are routed to their various destinations using Log Router Sinks.

Screenshot of Console Log Router with View sink details highlighted

Included in the settings are Exclusion filters, which define entries that can simply be discarded.

Screenshot of Console Log Router Sink Details popup showing exclusion filter

The solution

Make sure no exclusion filter is set for resource.type="cloud_functions"

Database connections

There are a number of issues that can arise when connecting to a database, many associated with exceeding connection limits or timing out. If you see a Cloud SQL warning in your logs, for example, "context deadline exceeded", you might need to adjust your connection configuration. See the Cloud SQL docs for additional details.