Troubleshooting Cloud Functions

This document shows you some of the common problems you might run into and how to deal with them.

Deployment

The deployment phase is a frequent source of problems. Many of the issues you might encounter during deployment are related to roles and permissions. Others have to do with incorrect configuration.

User with Viewer or Browser role cannot deploy a function

A user who has been assigned the Project Viewer, Project Browser, or Cloud Functions Viewer role has read-only access to functions and function details. These roles are not allowed to deploy new functions.

The error message

Cloud Console

You need permissions for this action. Required permission(s): cloudfunctions.functions.create

Cloud SDK

ERROR: (gcloud.functions.deploy) PERMISSION_DENIED: Permission
'cloudfunctions.functions.sourceCodeSet' denied on resource
'projects/<PROJECT_ID>/locations/<LOCATION>` (or resource may not exist)

The solution

Assign the user a role that has the appropriate access.

User with Project Viewer or Cloud Function role cannot deploy a function

In order to deploy a function, a user who has been assigned the Project Viewer, the Cloud Function Developer, or Cloud Function Admin role must be assigned an additional role.

The error message

Cloud Console

User does not have the iam.serviceAccounts.actAs permission on
<PROJECT_ID>@appspot.gserviceaccount.com required to create function.
You can fix this by running
'gcloud iam service-accounts add-iam-policy-binding <PROJECT_ID>@appspot.gserviceaccount.com --member=user: --role=roles/iam.serviceAccountUser'

Cloud SDK

ERROR: (gcloud.functions.deploy) ResponseError: status=[403], code=[Forbidden],
message=[Missing necessary permission iam.serviceAccounts.actAs for <USER>
on the service account <PROJECT_ID>@appspot.gserviceaccount.com. Ensure that
service account <PROJECT_ID>@appspot.gserviceaccount.com is a member of the
project <PROJECT_ID>, and then grant <USER> the role 'roles/iam.serviceAccountUser'.
You can do that by running
'gcloud iam service-accounts add-iam-policy-binding <PROJECT_ID>@appspot.gserviceaccount.com --member=<USER> --role=roles/iam.serviceAccountUser'
In case the member is a service account please use the prefix 'serviceAccount:' instead of 'user:'.]

The solution

Assign the user an additional role, the Service Account User IAM role (roles/iam.serviceAccountUser), scoped to the Cloud Functions runtime service account.

Deployment service account missing the Service Agent role when deploying functions

The Cloud Functions service uses the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) when performing administrative actions on your project. By default this account is assigned the Cloud Functions cloudfunctions.serviceAgent role. This role is required for Cloud Pub/Sub, IAM, Cloud Storage and Firebase integrations. If you have changed the role for this service account, deployment fails.

The error message

Cloud Console

Missing necessary permission resourcemanager.projects.getIamPolicy for
serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com on project <PROJECT_ID>.
Please grant serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
the roles/cloudfunctions.serviceAgent role. You can do that by running
'gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com --role=roles/cloudfunctions.serviceAgent'

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=7,
message=Missing necessary permission resourcemanager.projects.getIamPolicy
for serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
on project <PROJECT_ID>. Please grant
serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com
the roles/cloudfunctions.serviceAgent role. You can do that by running
'gcloud projects add-iam-policy-binding <PROJECT_ID> --member=serviceAccount:service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com --role=roles/cloudfunctions.serviceAgent'

The solution

Reset this service account to the default role.

Deployment service account missing Pub/Sub permissions when deploying a background function

The Cloud Functions service uses the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) when performing administrative actions. By default this account is assigned the Cloud Functions cloudfunctions.serviceAgent role. To deploy background functions, the Cloud Functions service must access Cloud Pub/Sub to configure topics and subscriptions. If the role assigned to the service account is changed and the appropriate permissions are not otherwise granted, the Cloud Functions service cannot access Cloud Pub/Sub and the deployment fails.

The error message

Cloud Console

Failed to configure trigger PubSub projects/<PROJECT_ID>/topics/<FUNCTION_NAME>

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=13,
message=Failed to configure trigger PubSub projects/<PROJECT_ID>/topics/<FUNCTION_NAME>

The solution

You can:

• Reset this service account to the default role.

  or

• Grant the pubsub.subscriptions.* and pubsub.topics.* permissions to your service account manually.

User missing permissions for runtime service account while deploying a function

In environments where multiple functions are accessing different resources, it is a common practice to use per-function identities, with named runtime service accounts rather than the default runtime service account ([e-mail]).

However, to use a non-default runtime service account, the deployer must have the iam.serviceAccounts.actAs permission on that non-default account. A user who creates a non-default runtime service account is automatically granted this permission, but other deployers must have this permission granted by a user with the correct permissions.

The error message

Cloud SDK

ERROR: (gcloud.functions.deploy) ResponseError: status=[400], code=[Bad Request],
message=[Invalid function service account requested: <SERVICE_ACCOUNT_NAME@<PROJECT_ID>.iam.gserviceaccount.com]

The solution

Assign the user the roles/iam.serviceAccountUser role on the non-default runtime service account. This role includes the iam.serviceAccounts.actAs permission.

Runtime service account missing project bucket permissions while deploying a function

Cloud Functions can only be triggered by events from Cloud Storage buckets in the same Google Cloud Platform project. In addition, the Cloud Functions Service Agent service account (service-<PROJECT_NUMBER>@gcf-admin-robot.iam.gserviceaccount.com) needs a cloudfunctions.serviceAgent role on your project.

The error message

Cloud Console

Deployment failure: Insufficient permissions to (re)configure a trigger
(permission denied for bucket <BUCKET_ID>). Please, give owner permissions
to the editor role of the bucket and try again.

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=7, message=Insufficient
permissions to (re)configure a trigger (permission denied for bucket <BUCKET_ID>).
Please, give owner permissions to the editor role of the bucket and try again.

The solution

You can:

• Reset this service account to the default role.

  or

• Grant the runtime service account the cloudfunctions.serviceAgent role.

  or

• Grant the runtime service account the storage.buckets.{get, update} and the resourcemanager.projects.get permissions.

Function deployment fails due to Cloud Build not supporting VPC-SC

Cloud Functions uses Cloud Build to build your source code into a runnable container. In order to use Cloud Functions with VPC Service Controls, you must configure an access level for the Cloud Build service account in your service perimeter.

The error message

Cloud Console

One of the below:

Error in the build environment

OR

Unable to build your function due to VPC Service Controls. The Cloud Build
service account associated with this function needs an appropriate access
level on the service perimeter. Please grant access to the Cloud Build
service account: '{PROJECT_NUMBER}@cloudbuild.gserviceaccount.com' by following
the instructions at
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls#grant-build-access"

Cloud SDK

One of the below:

ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Error in
the build environment

OR

Unable to build your function due to VPC Service Controls. The Cloud Build
service account associated with this function needs an appropriate access
level on the service perimeter. Please grant access to the Cloud Build
service account: '{PROJECT_NUMBER}@cloudbuild.gserviceaccount.com' by
following the instructions at
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls#grant-build-access"

The solution

If your project's Audited Resources logs mention "Request is prohibited by organization's policy" in the VPC Service Controls section and have a Cloud Storage label, you need to grant the Cloud Build Service Account access to the VPC Service Controls perimeter.

Function deployment fails due to incorrectly specified entry point

Cloud Functions deployment can fail if the entry point to your code, that is, the exported function name, is not specified correctly.

The error message

Cloud Console

Deployment failure: Function failed on loading user code. Error message:
Error: please examine your function logs to see the error cause:
https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs

Cloud SDK

ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function
failed on loading user code. Error message: Please examine your function
logs to see the error cause:
https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs

The solution

Your source code must contain an entry point function that has been correctly specified in your deployment, either via Cloud Console or Cloud SDK.

User with Project Editor role cannot make a function public

To ensure that unauthorized developers cannot modify authentication settings for function invocations, the user or service that is deploying the function must have the cloudfunctions.functions.setIamPolicy permission.

The error message

Cloud SDK

ERROR: (gcloud.functions.add-iam-policy-binding) ResponseError: status=[403], code=[Forbidden], message=[Permission 'cloudfunctions.functions.setIamPolicy' denied on resource 'projects/<PROJECT_ID>/locations/<LOCATION>/functions/<FUNCTION_NAME> (or resource may not exist).]

The solution

You can:

• Assign the deployer either the Project Owner or the Cloud Functions Admin role, both of which contain the cloudfunctions.functions.setIamPolicy permission.

  or

• Grant the permission manually by creating a custom role.

Serving

The serving phase can also be a source of errors.

Serving permission error due to the function being private

Cloud Functions allows you to restrict access to functions to end users and service accounts with the appropriate permission. This error message indicates that the caller does not have permission to invoke the function.

Note that deployed functions are by default set to be private.

The error message

HTTP Error Response code: 403 Forbidden

HTTP Error Response body: Error: Forbidden Your client does not have permission to get URL /<FUNCTION_NAME> from this server.

The solution

You can:

• Allow public (unauthenticated) access to all users for the specific function.

  or

• Assign the user the Cloud Functions Invoker Cloud IAM role for all functions.

Serving permission error due to "only allow internal traffic" configuration

Ingress settings restrict whether an HTTP function can be invoked by resources outside of your Google Cloud project or VPC Service Controls service perimeter. When the "Allow internal traffic only" setting for ingress networking is configured, this error message indicates that only requests from VPC networks in the same project or VPC Service Controls perimeter are allowed.

The error message

HTTP Error Response code: 403 Forbidden

HTTP Error Response body: Error 403 (Forbidden) 403. That's an error. Access is forbidden. That's all we know.

The solution

You can:

• Ensure that the request is coming from your Google Cloud project or VPC Service Controls service perimeter.

  or

• Change the ingress settings to allow all traffic for the function.

Logging

Setting up logging to help you track down problems can cause problems of its own.

Logs entries have no, or incorrect, log severity levels

Cloud Functions includes simple runtime logging by default. Logs written to stdout or stderr appear automatically in the Cloud Console. But these log entries, by default, contain only simple string messages.

The error

No or incorrect severity levels in logs.

The solution

To include log severities, you must send a structured log entry instead.

Note: As of September 2020, every newly released runtime will use the default severity log level behavior as defined by the native runtime library itself. So, for example, Python 3.8, the first of these runtimes, sets the default severity log level at WARNING or higher. If you wish to capture events with DEBUG or INFO severity using Python 3.8, you must also programmatically set your log level to include those levels.

Handle or log exceptions differently in the event of a crash

You may want to customize how you manage and log crash information.

The solution

Wrap your function is a try/catch block to customize handling exceptions and logging stack traces.

Note: Doing this may result in an unintended side effect in Background Functions which have set "retry on failure". Retrying failed events may itself fail.

Example

import logging
import traceback
def try_catch_log(wrapped_func):
  def wrapper(*args, **kwargs):
    try:
      response = wrapped_func(*args, **kwargs)
    except Exception: 
      # Replace new lines with spaces so as to prevent several entries which
      # would trigger several errors.
      error_message = traceback.format_exc().replace('\n', '  ')
      logging.error(error_message)
      return 'Error';
    return response;
  return wrapper;


#Example hello world function
@try_catch_log
def python_hello_world(request):
  request_args = request.args

  if request_args and 'name' in request_args:
    1 + 's'
  return 'Hello World!'

Logs too large in Node.js 10+, Python 3.8, Go 1.13, and Java 11

The max size for a regular log entry in these runtimes is 105 KiB.

The solution

Make sure you send log entries smaller that this limit.

Cloud Functions logs are not appearing via Logs Router Sink

Log entries are routed to their various destinations using Logs Router Sinks.

Included in the settings are Exclusion filters, which define entries that can simply be discarded.

The solution

Make sure no exclusion filter is set for resource.type="cloud_functions"