Troubleshooting logging in GKE


This page shows you how to investigate and resolve GKE logging related issues.

If you need additional assistance, reach out to Cloud Customer Care.

Missing cluster logs in Cloud Logging

Verify logging is enabled in the project

  1. List enabled services:

    gcloud services list --enabled --filter="NAME=logging.googleapis.com"
    

    The following output indicates that logging is enabled for the project:

    NAME                    TITLE
    logging.googleapis.com  Cloud Logging API
    

    Optional: Check the logs in Logs Viewer to determine who disabled the API and when they disabled the API:

    protoPayload.methodName="google.api.serviceusage.v1.ServiceUsage.DisableService"
    protoPayload.response.services="logging.googleapis.com"
    
  2. If logging is disabled, enable logging:

    gcloud services enable logging.googleapis.com
    

Verify logging is enabled on the cluster

  1. List the clusters:

    gcloud container clusters list \
        --project=PROJECT_ID \
        '--format=value(name,loggingConfig.componentConfig.enableComponents)' \
        --sort-by=name | column -t
    

    Replace the following:

    • PROJECT_ID: your Google Cloud project ID.

    The output is similar to the following:

    cluster-1              SYSTEM_COMPONENTS
    cluster-2              SYSTEM_COMPONENTS;WORKLOADS
    cluster-3
    

    If the value for your cluster is empty, logging is disabled. For example, cluster-3 in this output has logging disabled.

  2. Enable cluster logging if set to NONE:

    gcloud container clusters update CLUSTER_NAME  \
        --logging=SYSTEM,WORKLOAD \
        --location=COMPUTE_LOCATION
    

    Replace the following:

Verify nodes in the node pools have Cloud Logging access scope

One of the following scopes is required for nodes to write logs to Cloud Logging:

  • https://www.googleapis.com/auth/logging.write
  • https://www.googleapis.com/auth/cloud-platform
  • https://www.googleapis.com/auth/logging.admin
  1. Check the scopes configured on each node pool in the cluster:

    gcloud container node-pools list --cluster=CLUSTER_NAME \
        --format="table(name,config.oauthScopes)" \
        --location COMPUTE_LOCATION
    

    Replace the following:

    Migrate your workloads from the old node pool to the newly created node pool and monitor the progress.

  2. Create new node pools with the correct logging scope:

    gcloud container node-pools create NODE_POOL_NAME \
        --cluster=CLUSTER_NAME \
        --location=COMPUTE_LOCATION \
        --scopes="gke-default"
    

    Replace the following:

Verify the node pool's service account has a role with the correct IAM permissions

The service account must have a role which contains the permission logging.logEntries.create to create logs.

  1. Find the service account for each node pool:

    gcloud container node-pools list \
        --cluster=CLUSTER_NAME \
        --format="table(name,config.serviceAccount)" \
        --location=COMPUTE_LOCATION
    

    Replace the following:

    The output is similar to the following:

    NAME          SERVICE_ACCOUNT
    default-pool  gke-cluster-sa@developer.gserviceaccount.com
    

    If the node pool is using the default Compute Engine service account, you can describe it using the following command. As a best practice, use a minimally-privileged custom service account for your node pools. The default Compute Engine service account contains more than the minimum permissions needed to run your clusters.

    gcloud compute project-info describe --format="table(defaultServiceAccount)"
    
  2. Verify IAM roles have sufficient permissions.

    View the permissions contained in roles that are granted to a specific service account:

    gcloud projects get-iam-policy PROJECT_ID \
        --flatten="bindings[]" \
        --filter="bindings.members=serviceAccount:SERVICE_ACCOUNT" \
        --format="table[box](bindings.role)"
    
  3. Grant a role that contains the logging.logEntries.create permission to the service account. You can use a predefined role or create a custom role.

Verify that Cloud Logging write API quotas have not been reached

Confirm that you have not reached API write quotas for Cloud Logging.

  1. Go to the Quotas page in the Google Cloud console.

    Go to Quotas

  2. Filter the table by "Cloud Logging API".

  3. Confirm that you have not reached any of the quotas.

What's next

If you need additional assistance, reach out to Cloud Customer Care.