This document gives troubleshooting guidance for identity and authorization issues.
Keeping gcloud anthos auth
up-to-date
You can avoid many common issues by verifying that the components of your
gcloud anthos auth
installation are up-to-date.
There are two pieces that have to be verified, because the gcloud anthos auth
command has logic in the gcloud
core component and a separately packaged
anthos-auth
component.
Update gcloud
:
gcloud components update
Update anthos-auth
:
gcloud components install anthos-auth
Invalid provider configuration
If your identity provider configuration is invalid, you will see an error screen from your identity provider after you click LOGIN. Follow the provider-specific instructions to correctly configure the provider or your cluster.
Invalid permissions
If you complete the authentication flow, but still don't see the details of the cluster, make sure you granted the correct RBAC permissions to the account that you used with OIDC. Note that this might be a different account from the one you use to access Google Cloud console.
Missing refresh token
The following issue occurs when the authorization server prompts for consent, but the required authentication parameter wasn't provided.
Error: missing 'RefreshToken' field in 'OAuth2Token' in credentials struct
To resolve this issue, in your cluster configuration file, add prompt=consent
to the authentication.oidc.extraParams
field. Then regenerate the client
authentication file.
Refresh token expired
This issue occurs when the refresh token in the kubeconfig file has expired:
Unable to connect to the server: Get {DISCOVERY_ENDPOINT}: x509: certificate signed by unknown authority
To resolve this issue, run the login
command again.
gkectl create-login-config fails to get clientconfig
This issue occurs when the kubeconfig file passed to
gkectl create-login-config
is not for a user cluster or the ClientConfig
custom resource did not come up during cluster creation.
Error getting clientconfig using KUBECONFIG
To resolve this issue, make sure you have the correct kubeconfig file for your user cluster. Then check to see whether the ClientConfig object is in the cluster:
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get clientconfig default -n kube-public
gkectl create-login-config fails because of duplicate cluster name
This issue occurs if you attempt to write login configuration data that contains a cluster name that already exists in the destination file. Each login configuration file must contain unique cluster names.
error merging with file MERGE_FILE because MERGE_FILE contains a cluster with the same name as the one read from KUBECONFIG. Please write to a new output file
To resolve this issue, use the --output
flag to specify a new destination
file.
If you do not provide --output
, this login configuration data is written to a
file named kubectl-anthos-config.yaml
in the current directory.
gcloud anthos auth login fails with proxyconnect tcp
This issue occurs when there is an error in the https_proxy
or HTTPS_PROXY
environment variable configurations. If there's an https://
specified in the
environment variables, then the GoLang HTTP client libraries might fail if the
proxy is configured to handle HTTPS connections using other protocols such as
SOCK5.
Possible error message:
proxyconnect tcp: tls: first record does not look like a TLS handshake
To resolve this issue, modify the https_proxy
and HTTPS_PROXY
environment
variables to omit the https:// prefix
. On Windows, modify the system
environment variables. For example, change the value of the https_proxy
environment variable from https://webproxy.example.com:8000
to
webproxy.example.com:8000
.
Cluster access fails when using kubeconfig generated by gcloud anthos auth login
This issue occurs when the Kubernetes API server is unable to authorize the user. This can happen if the appropriate RBACs are missing or incorrect, or there is an error in the OIDC configuration for the cluster.
Unauthorized
To resolve this issue:
In the kubeconfig file generated by
gcloud anthos auth login
, copy the value ofid-token
.kind: Config … users: — name: … user: auth-provider: config: id-token: xxxxyyyy
Install jwt-cli and run:
jwt ID_TOKEN
Verify OIDC configuration.
The
authentication.oidc
in the user cluster configuration file has thegroup
andusername
fields, which are used to set the--oidc-group-claim
and--oidc-username-claim
flags in the Kubernetes API server. When the API server is presented with the token, it forwards the token to Anthos Identity Service, which returns the extractedgroup-claim
andusername-claim
back to the AIP server. The API server uses the response to verify that the corresponding group or user has the correct permissions.Verify that the claims set for
group
anduser
in theauthentication.oidc
section of cluster configuration file are present in the ID token.Check RBACs that were applied.
Verify that there is an RBAC with the correct permissions for either the user specified by
username-claim
or one of the groups listedgroup-claim
from the previous step. The name of the user or group in the RBAC should be prefixed with theusernameprefix
orgroupprefix
that was specified in the user cluster configuration file.Note that if
usernameprefix
is blank, andusername
is a value other thanemail
, the prefix defaults toissuerurl#
. To disable username prefixes, setusernameprefix
to-
.For more information about user and group prefixes, see Authenticating with OIDC,
Note that the Kubernetes API server treats a backslash as an escape character. Therefore, if the name of the user or group contains
\\
, the API server reads it as a single\
when parsing the ID token. Therefore, the RBAC role binding applied for this user or group should only contain a single backslash, or you might see anUnauthorized
error.Cluster configuration file:
oidc: ... username: "unique_name" usernameprefix: "-" group: "group" groupprefix: "oidc:"
ID token:
{ ... "email": "cluster-developer@example.com", "unique_name": "EXAMPLE\\cluster-developer", "group": [ "Domain Users", "EXAMPLE\\developers" ], ... }
The following RBAC bindings grant this group and user the
pod-reader
cluster role. Note the single slash in the name field instead of a double slash:Group ClusterRoleBinding:
apiVersion: kind: ClusterRoleBinding metadata: name: example-binding subjects: — kind: Group name: "oidc:EXAMPLE\developers" apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: pod-reader apiGroup: rbac.authorization.k8s.io
User ClusterRoleBinding:
apiVersion: kind: ClusterRoleBinding metadata: name: example-binding subjects: — kind: User name: "EXAMPLE\cluster-developer" apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: pod-reader apiGroup: rbac.authorization.k8s.io
Check the Kubernetes API server logs.
If the OIDC plugin configured in the Kubernetes API server does not start up correctly, the API server returns an
Unauthorized
error when presented with the ID token. To see if there were any issues with the OIDC plugin in the API server, run:kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG logs statefulset/kube-apiserver -n USER_CLUSTER_NAME