Troubleshoot common issues

This page shows you how to resolve common issues with GKE on Azure.

If you need additional assistance, reach out to Cloud Customer Care.

Common error messages

The following sections explain the causes and resolutions for some common error messages.

Server doesn't have a resource

Errors such as error: the server doesn't have a resource type "services" can happen when a cluster has no running node pools, or Connect gateway cannot connect to a node pool. To check the status of your node pools, run the following command:

gcloud container azure node-pools list \
    --cluster-name CLUSTER_NAME \
    --location LOCATION

Replace the following:

  • CLUSTER_NAME: your cluster's name
  • LOCATION: the Google Cloud location that manages your cluster

The output includes the status of your cluster's node pools. If you don't have a node pool listed, Create a node pool.

Forbidden user

The following error occurs when your username does not have administrator access to your cluster:

Error from server (Forbidden): users "administrator@example.com" is forbidden:
User "system:serviceaccount:gke-connect:connect-agent-sa" cannot impersonate
resource "users" in API group "" at the cluster scope

You can configure additional users by passing the --admin-users flag when you create a cluster.

If you use Connect gateway and can't connect to your cluster, try the following steps:

  1. Get the authorized users for your cluster.

    gcloud container azure clusters describe CLUSTER_NAME \
    --format 'value(authorization.admin_users)'

    Replace CLUSTER_NAME with your cluster's name.

    The output includes the usernames with administrator access to the cluster. For example:

    {'username': 'administrator@example.com'}

  2. Get the username currently authenticated with the Google Cloud CLI.

    gcloud config get-value account

    The output includes the account authenticated with the Google Cloud CLI. If the output of the gcloud containers azure clusters describe and gcloud config get-value account don't match, run gcloud auth login and authenticate as the username with administrative access to the cluster.

Issues with kubectl commands

The following sections provide guidance on how to resolve issues with unresponsive or failing kubectl commands.

kubectl commands stop responding

If your cluster runs a Kubernetes version earlier than 1.25 and kubectl commands are unresponsive or time out, the most common reason is that you have not yet created a node pool. By default, GKE on Azure generates kubeconfig files that use Connect gateway as an internet-reachable endpoint. For this to work, the gke-connect-agent Deployment needs to be running in a node pool on the cluster.

For more diagnostic information, run the following command:

kubectl cluster-info -v=9

If there are no running node pools, you see requests to connectgateway.googleapis.com fail with a 404 cannot find active connections for cluster error.

For clusters with a Kubernetes version of 1.25 or later, the gke-connect-agent runs on the control plane, and a node pool is not required. If the kubectl command is unresponsive, check the control plane component logs with Cloud Logging.

kubectl exec, attach, and port-forward commands fail

The kubectl exec, kubectl attach, and kubectl port-forward commands might fail with the message error: unable to upgrade connection when using Connect gateway. This is a limitation when using Connect gateway as your Kubernetes API Server endpoint.

To work around this, use a kubeconfig that specifies the cluster's private endpoint. For instructions on accessing the cluster through its private endpoint, see Configure cluster access for kubectl.

Generic kubectl troubleshooting

If you use Connect gateway:

  • Ensure you have enabled Connect gateway in your Google Cloud project:

    gcloud services enable connectgateway.googleapis.com

  • For clusters with a Kubernetes version earlier than 1.25, ensure that you have at least one Linux node pool running and that the gke-connect-agent is running. For details, see Troubleshoot cluster connections.

  • For clusters with a Kubernetes version of 1.25 or later, check the gke-connect-agent logs with Cloud Logging.

What's next