Troubleshoot Eventarc for Cloud Run

This page shows you how to resolve issues that you might encounter when using Eventarc.

Eventarc trigger provider

This section provides troubleshooting tips for issues at the source of events (the provider).

Provider doesn't exist

Check that you have specified the trigger filters appropriately.

Events are produced from the provider, but are not received at the target

Check that you are filtering for events from providers that are in the same region as the Cloud Run target. For more information, see locations supported by Eventarc.

A trigger that previously delivered events has stopped working

  1. Verify that the provider is generating events. Check the Cloud Audit Logs and make sure the monitored service is writing logs. If logs are recorded but events are not delivered, contact support.

  2. Verify that a Pub/Sub topic with the same trigger name exists.

    1. To list triggers, see gcloud eventarc triggers list.

    2. To list the Pub/Sub topics, run:

        gcloud pubsub topics list

    Verify that the Pub/Sub topic name includes the name of the created trigger. If the Pub/Sub topic is missing, create a topic when you create the trigger.

  3. Verify the health of the Pub/Sub topic:

    1. On the Cloud console, go to Cloud Run, select the service you created and navigate to the Triggers tab. Check that there is a checkmark on the Triggers tab.

    2. Verify the Cloud Pub/Sub topic configuration. If the Pub/Sub topic is missing, create the topic when you create the trigger. If the problem persists, contact support.

      Go to Pub/Sub topics

    3. Monitor if messages are published to the topic with the metric: topic/send_message_operation_count. If messages are not published to the topic, check Cloud Audit Logs and make sure the monitored service is emitting logs. If logs are recorded but events are not delivered, contact support.

    4. Monitor if messages are pushed successfully with the metric: subscription/push_request_count by response_code. If push errors are reported, check the Cloud Run service logs. If the receiving endpoint returns a non-OK status code, it indicates that the Cloud Run code is not working as expected and you must contact support.

Eventarc trigger

This section provides troubleshooting tips for issues at the trigger.

Failed to load service details: permission denied

When you create the trigger, check if the trigger permissions are configured, and the Pub/Sub service account is granted the iam.serviceAccountTokenCreator role. For more information, see Eventarc roles and permissions.

Unexpected charges when using Eventarc

When you create a trigger, check if the filters configured could lead to triggers being set off indefinitely. To help avoid unexpected charges on your bill, set budgets and alerts using Google Cloud's billing console. For more information, see Create a budget.

Eventarc trigger target

This section provides troubleshooting tips for issues at the target.

Cloud Run service is not receiving events

If a service does not receive events, here are possible resolutions per event type:

Cloud Audit Logs
  • Check that the Admin Read, Data Read, and Data Write log types are enabled. Cloud Audit Logs are enabled under IAM & Admin > Audit Logs.
  • Check that enough time has elapsed since creating the trigger. Audit log triggers can take up to two minutes before sending events.
  • Follow the prerequisites for Eventarc trigger creation.
  • Check that your event filters are not too restrictive. For more information, see event filters for Cloud Audit Logs.
  • Check audit logs under Logging > Logs Explorer to confirm that the monitored service is generating audit logs.
  • Check Cloud Run logs under Cloud Run: Click the service you created in Cloud Run and navigate to the Logs tab. If the logs show that the service received a non-OK status code, the Cloud Run service is rejecting events.
Pub/Sub
  • Confirm that the Eventarc trigger is created.
  • Confirm that the Pub/Sub topic is created and attached to the Eventarc trigger.
  • Check that the Pub/Sub topic exists: Go to Pub/Sub and search for the topic.
  • Check Cloud Run logs under Cloud Run: Click the service you created in Cloud Run and navigate to the Logs tab. If the logs show that the service received a non-OK code, the Cloud Run service is rejecting events.

Trigger is set, but target does not receive events

  • Check that the event payload you are sending does not exceed the limits on event size (512KB). The Pub/Sub push subscriptions are also subject to a set of resource limits such as throughput per publish region. This makes sure that the Pub/Sub event payload is less than 512KB.

  • Locate the Pub/Sub topic associated with the Cloud Run trigger.

    1. Go to Pub/Sub > Topics.

      Go to Pub/Sub topics

    2. Verify that the Pub/Sub topic name includes the name of the created trigger.

  • Monitor if messages are published to the topic with the metric: topic/send_message_operation_count. If messages are not published to the topic, check Cloud Audit Logs and make sure the monitored service is emitting logs. If logs are recorded but events are not delivered, contact support.

  • Monitor if messages are pushed successfully with the metric: subscription/push_request_count by response_code. If push errors are reported, check the Cloud Run service logs. If the receiving endpoint returns a non-OK status code, it indicates that the Cloud Run code is not working as expected and you must contact support.

Cannot find your Pub/Sub topic

The Pub/Sub topic could have been removed or modified. A new Pub/Sub topic is created when you create a trigger, so delete the existing trigger and create it again.

INVALID_ARGUMENT: cloud run service "SERVICE_NAME" in region "REGION_NAME" does not exist

The service account has been deleted. Events will not be delivered to Cloud Run if the service account for the target service is deleted. Create a new Cloud Run service account.

Cloud Run service is down

Even if the Cloud Run service is down, Pub/Sub retains events for seven days by default and will retry sending events to the target. For more information on Pub/Sub resource limits, see Quotas and limits.

You can ensure that events sent from Pub/Sub to the target are not dropped by setting up a Pub/Sub subscription retry policy or forwarding undelivered messages to a dead-letter topic (also known as a dead-letter queue).

Before setting the dead-letter topic, retrieve the trigger's topic and subscription:

gcloud eventarc triggers describe TRIGGER \
    --location=LOCATION

Replace the following:

  • TRIGGER: the ID of the trigger or a fully qualified identifier.
  • LOCATION: the location of the Eventarc trigger.