Troubleshooting an Amazon Managed Streaming for Apache Kafka import topic

This document provides some common troubleshooting tips for Amazon Managed Streaming for Apache Kafka (Amazon MSK) import topics.

After you configure the import topic and start ingesting messages, you can check the relevant Cloud Monitoring metrics to see if data is ingested. Perform the following steps:

  1. In the console, go to the Topics page.

    Go to Topics

  2. Click the Amazon MSK import topic that you want to troubleshoot.

  3. In the topic details page, click the Metrics tab.

  4. Check the chart for the metric Ingestion byte count.

  5. If no data is being ingested, in the topic details page, check to see if there is an error for the Topic state field.

  6. You can also check the metric Ingestion data source state. To do so, in the topic details page, click the Metrics tab.

  7. Here is a list of errors that you might encounter:

    Error Code Description Fix
    MSK_PERMISSION_DENIED Error consuming Amazon MSK data due to permission issues. Verify the AWS role ARN for accuracy and check that the AWS role has the required read permissions. For more information, see Create a policy in AWS.
    Ensure that a service account exists and that it is correctly configured as described in the procedure Create a role in AWS using a custom trust policy. For more information about how to create a service account, see Create a service account in Google Cloud.
    Verify that the Pub/Sub service account has the iam.serviceAccounts.getOpenIdToken permission. For more information, see Add the Service Account Token Creator role to the Pub/Sub service account.
    Verify that you have added the service account user role to the service account. For more information, see Add the service account user role to the service account.
    PUBLISH_PERMISSION_DENIED Error publishing to the Pub/Sub topic due to permission issues. Grant the Pub/Sub service account the necessary publish permissions. For more information, see Add the Pub/Sub publisher role to the Pub/Sub service account.
    CLUSTER_NOT_FOUND The provided cluster wasn't found. Verify the cluster ARN for accuracy. Ensure that the cluster exists and is accessible.
    TOPIC_NOT_FOUND The provided Kafka topic wasn't found. Verify the topic name for accuracy. Ensure that the topic exists within the specified cluster.

If there are no errors till this point, check the best practices in Maintain a healthy publisher.

You can also enable platform logs for an import topic. These logs help you troubleshoot issues when you have issues ingesting objects into Pub/Sub import topics.

Apache Kafka® is a registered trademark of The Apache Software Foundation or its affiliates in the United States and/or other countries.