This page shows you how to resolve issues that you might encounter when using Eventarc for GKE, including the public endpoints of private and public services running in a GKE cluster.
For other issues that you might also encounter, refer to troubleshooting for:
Trigger creation fails with resource creation permission errors
You receive error messages similar to one of the following:
Error applying IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT': Error setting IAM policy for service account 'projects/PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT': googleapi: Error 403: Permission iam.serviceAccounts.setIamPolicy is required to perform this operation on service account SERVICE_ACCOUNT., forbidden
Error when reading or editing Container Cluster "cluster": googleapi: Error 403: Required "container.clusters.get" permission(s) for "..."., forbidden
Error reading instance group manager returned as an instance group URL: "googleapi: Error 403: Required 'compute.instanceGroupManagers.get' permission for '...', forbidden"
This error occurs when you have not correctly set up the Eventarc service account that is required by Eventarc to manage resources in the Google Kubernetes Engine (GKE) cluster.
To resolve the issue, ensure that the Eventarc service account has been configured correctly with the appropriate permissions to create resources. For more information, follow the instructions for a specific provider, event type, and GKE destination.
Trigger creation fails because target cluster cannot be found or does not have Workload Identity Federation for GKE enabled
You receive error messages similar to one of the following:
ERROR: (gcloud.eventarc.triggers.create) NOT_FOUND: Not found: projects/PROJECT_ID/locations/ LOCATION/clusters/CLUSTER_NAME. ...
ERROR: (gcloud.eventarc.triggers.create) INVALID_ARGUMENT: workload identity is not enabled on cluster CLUSTER_NAME in LOCATION ...
This error occurs when Eventarc cannot find the target GKE cluster or the cluster does not have Workload Identity Federation for GKE enabled.
To resolve the issue, ensure that the target cluster exists and has Workload Identity enabled.
Trigger creation fails because of invalid resource state
You receive an error message similar to the following for your project:
ERROR: (gcloud.eventarc.triggers.create) Invalid resource state for projects/PROJECT_ID/locations/LOCATION/triggers/TRIGGER_ID. Ensure that you have enabled your GKE destination and assigned required permissions to the service account. For more information, see https://cloud.google.com/eventarc/standard/docs/gke/troubleshooting.
This error occurs when:
- Your GKE destination hasn't been enabled.
- The service account has not been granted the required roles and permissions that enables Eventarc to manage events for GKE destinations.
To resolve this issue:
- Ensure that you have enabled GKE destinations:
gcloud eventarc gke-destinations init
- Ensure that the Eventarc service account has been configured with the appropriate permissions to create resources. For more information on how to grant the appropriate roles to the service account, follow the instructions for a specific provider and event type in the "Prepare to create a trigger" section.
- If the error persists, contact support.
Trigger is created successfully but target is not receiving events
Check that enough time has elapsed since creating the trigger. It can take up to two minutes before events are sent.
If events are not being delivered at the target, events sent from Pub/Sub to the target might be getting dropped:
To ensure that events aren't dropped, set up a Pub/Sub subscription retry policy or forward undelivered messages to a dead-letter topic (also known as a dead-letter queue).
Before setting the dead-letter topic, retrieve the trigger's topic and subscription:
gcloud eventarc triggers describe TRIGGER \ --location=LOCATION
Replace the following:
TRIGGER
: the ID of the trigger or a fully qualified identifier.LOCATION
: the location of the Eventarc trigger.
If the trigger still doesn't work, check the status of Eventarc's event forwarder component to see if it has been deployed properly:
- Identify the cluster namespaces by listing all pods that have
event-forwarder
in their name: The output is similar to the following:kubectl get pods --all-namespaces | grep event-forwarder
The first 30 characters of the namespaces should match the name of the Eventarc trigger you created, followed by a hash to disambiguate any triggers with the same initial 30 characters.event-forwarder-namespace-hash1 event-forwarder-hash1 1/1 Running 0 5d2h event-forwarder-namespace-hash2 event-forwarder-hash2 1/1 Running 0 2d2h
- Retrieve the deployment details:
kubectl get deployments -n EVENT_FORWARDER_NAMESPACE event-forwarder
- Retrieve the pod details:
kubectl describe pod -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
- Print the pod logs to investigate why or where the failure has occurred:
For example, in the following log, the service account has had thekubectl logs -f -n EVENT_FORWARDER_NAMESPACE EVENT_FORWARDER_HASH
Pub/Sub Subscriber
role revoked:{"error":"generic::permission_denied: missing 'pubsub.subscriptions.consume' permission", "message":"Permissions check failed","severity":"fatal","timestamp":"2021-07-07T14:57:49.038877483Z"}
- Identify the cluster namespaces by listing all pods that have
Related information
- Learn more about Eventarc for GKE.
- To get additional help, see Get support.