Troubleshooting a Cloud Storage import topic

This document provides some common troubleshooting tips for Cloud Storage import topics.

After you configure the import topic and start ingesting messages, you can check the relevant Cloud Monitoring metrics to see if data is ingested. Perform the following steps:

  1. In the console, go to the Topics page.

    Go to Topics

  2. Click the Cloud Storage import topic that you want to troubleshoot.

  3. In the topic details page, click the Metrics tab.

  4. Check the chart for the metric Ingestion byte count.

  5. If no data is being ingested, in the topic details page, check to see if there is an error and a red exclamation mark for the Topic state field.

  6. You can also check the metric Ingestion data source state. To do so, in the topic details page, click the Metrics tab.

    For more information about these metrics, see Metrics to monitor import topics.

  7. Here is a list of errors that you might encounter:

    Error Code Description Fix
    CLOUD_STORAGE_PERMISSION_DENIED Error consuming Cloud Storage data due to permission issues. Verify that the Pub/Sub service account has the necessary permissions to access the Cloud Storage bucket. Ensure the service account has the following permissions:
    • storage.objects.list: To list the objects in a bucket.
    • storage.objects.get: To read the objects in a bucket.
    • storage.buckets.get: To obtain the bucket location and that the bucket exists.

    For more information, see the prerequisites to creating a Cloud Storage topic.
    PUBLISH_PERMISSION_DENIED Error publishing to the topic due to permission issues. Grant the Pub/Sub service account the necessary publish permissions. For more information, see Add the Pub/Sub publisher role to the Pub/Sub service account.
    BUCKET_NOT_FOUND The specified Cloud Storage bucket cannot be found. Verify the bucket name for accuracy.
    TOO_MANY_OBJECTS The Cloud Storage bucket has too many objects, ingestion is paused. Ingestion is resumed after the number of objects in the bucket is fewer than 50 million. The limit for the objects is 50 million. Reduce the number of objects in the bucket or contact Google Cloud support for assistance.

If there are no errors till this point, check the best practices in Maintain a healthy publisher.

You can also enable platform logs for a Cloud Storage import topic. These logs help you troubleshoot issues when you have issues ingesting objects into Pub/Sub import topics.