Troubleshooting fleet-level setup issues
This document provides troubleshooting instructions for fleet-level setup of GKE Identity Service.
Resolve fleet-level setup issues
To resolve issues that may occur while setting up your fleet, follow these instructions:
1. Validate if GKE Identity Service is enabled
To validate if GKE Identity Service is enabled for your project, run the following command:
$ gcloud container fleet identity-service describe
If you don't see any errors, GKE Identity Service is enabled and you can continue to the next instruction.
Error:
Identity Service Feature for project <your-project-id> is not enabled
If you see this error, it indicates that GKE Identity Service was not enabled correctly for your fleet when setting up the feature.
Solution: To ensure the feature is enabled for your project, see the instructions in Configure clusters for GKE Identity Service.
2. View GKE Identity Service status
To view the current status of GKE Identity Service, run the following command:
$ gcloud container fleet identity-service describe
The output of this command has the following two sections:
membershipSpecs
contains the identity provider configuration that you've specified for each cluster registered to the fleet. GKE Identity Service is configured on each cluster using these configurations.membershipStates
shows the current state of GKE Identity Service in each cluster, including the relevant identity provider configuration and any errors that might have occurred during setup.
To troubleshoot issues in a cluster, under membershipStates
, search for the cluster and look for the following fields:
state
field:membershipStates -> <cluster-identifier> -> identityservice -> state
failureReason
field:membershipStates -> <cluster-identifier> -> identityservice -> failureReason
.
Example
If you have two clusters helloworld
and foobar
registered to the fleet,
then the output has the following structure:
Identity Service Feature:
createTime: '222-11-10T23:05:6.146566392Z'
membershipSpecs:
projects/<your-project-number>/locations/<location>/memberships/helloworld:
identityservice:
authMethods:
- name: oidc
oidcConfig:
clientId: id
issuerUri: https://accounts.google.com
... other fields
projects/<your-project-number>/locations/<location>/memberships/foobar:
identityservice:
authMethods:
- name: oidc
oidcConfig:
clientId: id
issuerUri: https://accounts.google.com
... other fields
membershipStates:
projects/<your-project-number>/location/<location>/memberships/helloworld:
identityservice:
failureReason: AIS feature does not support this cluster type
memberConfig:
authMethods:
- name: oidc
oidcConfig:
clientId: id
issuerUri: https://accounts.google.com
... other fields
state: ERROR
state:
code: OK
updateTime: '2022-11-11T20:39:59.556176650Z'
projects/<your-project-number>/location/<location>/memberships/foobar:
identityservice:
memberConfig:
authMethods:
- name: oidc
oidcConfig:
clientId: id
issuerUri: https://accounts.google.com
... other fields
state: OK
state:
code: OK
updateTime: '2022-11-11T20:39:59.556176650Z'
name: projects/<project-number>/locations/global/features/identityservice
updateTime: '2022-11-11T20:40:31.859321901Z'
3. Verify cluster state
To verify the state of GKE Identity Service in a cluster, check the following:
state
value isOK
: This indicates that no error occurred when configuring GKE Identity Service on the cluster with the configuration specified undermembershipSpecs
.If the configuration for the cluster you are troubleshooting (for example,
helloworld
) is different inmembershipSpecs
andmembershipStates
, then it’s likely that you updated the configuration for the cluster. Wait a few minutes for GKE Identity Service to propagate the changes to the cluster and check the status again.If the configuration for the cluster you are troubleshooting (for example,
helloworld
) is the same inmembershipSpecs
andmembershipStates
, then GKE Identity Service has been configured correctly.state
field isERROR
: This indicates that there were errors when configuring GKE Identity Service on the cluster with the configuration specified undermembershipSpecs
.To resolve the issue, see Troubleshoot common issues. After following the necessary troubleshooting steps, wait for a few minutes and then check the status again.
Troubleshoot common issues
GKE Identity Service API not enabled
This issue occurs when the GKE Identity Service API is not enabled.
Error message
anthosidentityservice.googleapis.com is not enabled
Solution
To enable the API, run the following command:
$ gcloud services enable anthosidentityservice.googleapis.com
Unsupported cluster type
This issue occurs when you use a cluster type that isn't supported.
Error message
GKE Identity Service feature does not support this cluster type
Solution
For information on the supported cluster types, see cluster types or reach out to Google Cloud support for new cluster type requests .
Unsupported protocols found in the configuration
This issue occurs when the identity configuration that you've applied to your cluster contains protocols that aren’t supported.
Error message
unsupported protocol found in configuration, aborting reconciliation.
Solution
Fleet-level setup for GKE Identity Service supports the following:
Update the configuration on the cluster so that it only contains the above supported protocols. To edit the configuration, run the following command:
kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-public edit clientconfig default
Replace CLUSTER_KUBECONFIG
with the path to the kubeconfig file for the cluster. If there are multiple contexts in the kubeconfig, the current context is used. You may need to reset the current context to the correct cluster before running the command.
Cluster does not have an identity configuration
This issue occurs when there is no configuration applied to your cluster.
Error message
Authentication configuration is not present for this membership
Solution
Apply an identity configuration to the cluster by following the instructions in Configure clusters.
Missing GKE Hub admin IAM permission
This issue occurs when you don't have the gkehub.admin
role on the project that you're using.
Error messages
PERMISSION_DENIED: Permission 'gkehub.memberships.list' denied on '`projects//locations/ /memberships`'
PERMISSION_DENIED: Permission 'gkehub.features.get' denied on '`projects//locations/global/features/identityservice`'
PERMISSION_DENIED: Permission 'gkehub.features.delete' denied on '`projects//locations/global/features/identityservice`'
PERMISSION_DENIED: Permission 'gkehub.features.create' denied on '`projects//locations/global/features/identityservice`'
Solution
An owner of the project with id project_id
must run the following command:
gcloud projects add-iam-policy-binding project_id --member=user:USER_ID --role=roles/gkehub.admin
Replace USER_ID
with your Google account.
Non-unique identity providers present in an identity configuration
This issue occurs when the GKE Identity Service configuration for a cluster
has the same name
value for multiple identity providers.
Error message
Configuration contains multiple identity providers with the same name
Solution
Rename or remove duplicate entries from your configuration.
Multiple providers of type google
present in an identity configuration
This issue occurs when the ClientConfig is configured with multiple configurations of type google
.
Error message
configuration contains multiple identity providers of type 'google'
Solution
The spec/authentication
section of the configuration can contain at most one configuration of type google
.
Update the configuration on the cluster so that it does not have more than one configuration of type google
. To edit the configuration, run the following command:
kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-public edit clientconfig default
Replace CLUSTER_KUBECONFIG
with the path to the kubeconfig file for the cluster. If there are multiple contexts in the kubeconfig, the current context is used. You may need to reset the current context to the correct cluster before running the command.
Error fetching the ClientConfig's custom resource definition
This issue occurs when there is an error fetching the ClientConfig's custom resource definition from the cluster.
Error message
error fetching the ClientConfig's Custom Resource Definition: ERROR_REASON
Solution
Use the details displayed in the ERROR_REASON
to resolve this issue.
Error updating the GKE Identity Service ClientConfig
This issue occurs when there is an error in updating the ClientConfig in the cluster.
Error message
Error updating AIS ClientConfig: ERROR_REASON
Solution
Use the details displayed in the ERROR_REASON>
to resolve this issue.
ClientConfig not present on the cluster
This issue occurs when the ClientConfig, which should be created when installing GKE Identity Service, is not present on the cluster. A missing ClientConfig indicates that GKE Identity Service has not been correctly installed on the cluster.
Error message
ClientConfig CR not present, expected to be created when AIS was installed
Solution
The GKE Identity Service resources on the cluster may have been accidentally deleted. You can try to create a new cluster. If you see the same error in the new cluster, contact your cluster administrator or Google Cloud support.
GKE Identity Service pod not started
This issue occurs when the GKE Identity Service pod is in a pending
state.
Error message
GKE Identity Service pod is not running (pending)
Solution
This issue should auto-resolve in a few minutes, after which the GKE Identity Service pod
switches to the running
state.