Set up clusters with GKE Identity Service at a fleet level

This document is for cluster administrators or application operators who want to set up GKE Identity Service on their clusters. It has instructions on how to set up GKE Identity Service at the fleet level on your clusters with your preferred identity provider.

Enable APIs

To get started, you must enable the relevant APIs.

Console

Ensure that the project where the clusters are registered is selected.

Enable the GKE Hub and Kubernetes Engine APIs.
Enable the APIs

gcloud

Run the following command to enable the required APIs for setup:

gcloud services enable 

gkehub.googleapis.com 

container.googleapis.com

Configure clusters

To configure your clusters to use your chosen provider, GKE Identity Service needs you to specify details about the identity provider, the information in the JWT tokens it provides for user identification, and other information provided when registering GKE Identity Service as a client application. So, for example, if your provider creates identity tokens with the following fields (among others), where iss is the identity provider URI, sub identifies the user, and groupList lists the security groups that the user belongs to:

{
  'iss': 'https://server.example.com'
  'sub': 'u98523-4509823'
  'groupList': ['developers@example.corp', 'us-east1-cluster-admins@example.corp']
  ...
}

...your configuration will have the following corresponding fields:

issueruri: 'https://server.example.com'
username: 'sub'
group: 'groupList'
...

Your platform administrator, or whoever manages identity in your organization, should provide you with most of the information you need to create the configuration. For example configurations for some commonly used identity providers, see Provider-specific configurations.

GKE Identity Service lets you create or update and apply this configuration from the Google Cloud console or using the Google Cloud CLI.

Console

Enable GKE Identity Service

In the Google Cloud console, go to the Feature Manager page.

Go to Feature Manager
Click Enable in the Identity Service panel, then click Enable again in the pane that displays. This creates a new GKE Identity Service controller instance to manage the lifecycle of GKE Identity Service in your fleet's clusters.

Select clusters

Back in the Feature Manager page, click Details in the Identity Service panel to open the details pane for the service. This displays your project's clusters and their fleet-level GKE Identity Service status.
Click Update identity service to open the setup pane.
Select the clusters you want to configure. Only supported cluster types can be selected. You can choose individual clusters, or specify that you want all clusters to be configured with the same identity configuration. If you have configured fleet-level defaults, the configuration is reconciled back to the default. For more information, see Configure fleet-level defaults.
In the Identity provider drop-down, choose how you want to configure the cluster. If the cluster has an existing GKE Identity Service configuration, you can choose to update it. If an existing registered cluster has an GKE Identity Service configuration you'd like to use, you can choose to copy that configuration to your selected clusters. To create a completely new configuration, follow the instructions for your chosen provider as described in the next section.

Set provider details

The provider details you need to add depend on the identity provider type you want to use for your configuration.

OIDC

Select New Open ID Connect to create a new OIDC configuration.
Specify the name you want to use to identify this configuration in the Provider name field, typically the identity provider name. This name must start with a letter followed by up to 39 lowercase letters, numbers, or hyphens, and cannot end with a hyphen. You cannot edit this name once you've created a configuration.
Specify the client ID returned when registering GKE Identity Service with your provider in the Client ID field.
Specify the client secret that must be shared between the client application and the identity provider in the Client Secret field.
Specify the URI where authorization requests are made to your identity provider in the Issuer URL field.
Click Next to set the OIDC attributes.

Azure AD

Select New Azure Active Directory to create a new Azure AD configuration.
Specify the name you want to use to identify this configuration in the Provider name field, typically the identity provider name. This name must start with a letter followed by up to 39 lowercase letters, numbers, or hyphens, and cannot end with a hyphen. You cannot edit this name once you've created a configuration.
Specify the client ID returned when registering GKE Identity Service with your provider in the Client ID field.
Specify the client secret that must be shared between the client application and the identity provider in the Client Secret field.
Specify the tenant that is the Azure AD account to be authenticated in the Tenant.
Click Next to set the Azure AD attributes.

LDAP

Select LDAP to create a new LDAP configuration.
Specify the name you want to use to identify this configuration in the Provider name field, typically the identity provider name. This name must start with a letter followed by up to 39 lowercase letters, numbers, or hyphens, and cannot end with a hyphen. You cannot edit this name once you've created a configuration.
Click Next.
Specify the hostname (mandatory), LDAP connection type, and base64-encoded CA certificate of the LDAP server.
Click Next to configure the server.
Specify the user's distinguished name, filter, login attribute, and identifier attribute.
Click Next to set the user details.
If you choose to use groups, specify the group's distinguished name, filter, and identifier attribute.
Click Next to set the group details.
Specify the service account username and password.
Click Done to set the service account name.

Set attributes

The attributes you need to add depend on your identity provider and the setup options chosen by your platform administrator when configuring the provider for GKE Identity Service.

OIDC

Fill in the configuration attributes:
- kubectl redirect URI: The redirect URL and port used by the gcloud CLI and specified by your platform administrator at registration, typically in the form http://localhost:PORT/callback.
- Certificate Authority (Optional): If provided by your platform administrator, a PEM-encoded certificate string for the identity provider.
- Group Claim (Optional): The JWT claim (field name) that your provider uses to return an account's security groups.
- Group Prefix (Optional): The prefix you want to prepend to security group names to avoid clashes with existing names in your access control rules if you have configurations for multiple identity providers (typically the provider name).
- Proxy (Optional): Proxy server address to use to connect to the identity provider, if applicable. You might need to set this if, for example, your cluster is in a private network and needs to connect to a public identity provider. For example: http://user:password@10.10.10.10:8888.
- Scopes (Optional): Any additional scopes required by your identity provider. Microsoft Azure and Okta require the offline_access scope. Click Add scope to add more scopes if necessary.
- User Claim (Optional): The JWT claim (field name) that your provider uses to identify an account. If you don't specify a value here, GKE Identity Service uses "sub", which is the user ID claim used by many providers. You can choose other claims, such as "email" or "name", depending on the OpenID provider. Claims other than "email" are prefixed with the issuer URL to prevent naming clashes.
- User Prefix (Optional): The prefix you want prepended to user claims to prevent clashes with existing names, if you don't want to use the default prefix.
- Extra Params (Optional): Any extra parameters required for your configuration, specified as the parameter Key and Value. Click Add param to add more parameters if needed.
- Enable access token (Optional): If enabled, it allows group support for OIDC providers such as Okta.
- Deploy Google Cloud console proxy (Optional): If enabled, a proxy is deployed that lets the Google Cloud console connect to an on-premises identity provider that is not publicly accessible over the internet.

Azure AD

Fill in the configuration attributes:
- kubectl redirect URI: The redirect URL and port used by the gcloud CLI and specified by your platform administrator at registration, typically in the form http://localhost:PORT/callback.
- User Claim (Optional): The JWT claim (field name) that your provider uses to identify an account. If you don't specify a value here, GKE Identity Service uses a value in the order of "email", "preferred_username", or "sub" to fetch the user details.
- Proxy (Optional): Proxy server address to use to connect to the identity provider, if applicable. You might need to set this if, for example, your cluster is in a private network and needs to connect to a public identity provider. For example: http://user:password@10.10.10.10:8888.

Add identity provider

If you have additional identity providers that you'd like to configure for your fleet, you have the option to add the providers here. Follow the steps to specify additional identity providers.

Update configuration

Click Update configuration. This installs GKE Identity Service if necessary (EKS clusters only, GKE clusters already have GKE Identity Service installed by default) and applies the client configuration on your selected clusters.

gcloud

Create the configuration file

GKE Identity Service uses a Kubernetes custom resource type (CRD) called ClientConfig for cluster configuration, with fields for all the information GKE Identity Service needs to interact with the identity provider. The following sections provide the configurations for OIDC and LDAP where you create a file called auth-config.yaml with your configuration.

OIDC

The following file shows both an oidc configuration and an azuread configuration. For more information on when to use oidc or azuread, see Provider-specific configurations.

  apiVersion: authentication.gke.io/v2alpha1
  kind: ClientConfig
  metadata:
    name: default
    namespace: kube-public
  spec:
    authentication:
    - name: NAME
      proxy: PROXY_URL
      oidc:
        certificateAuthorityData: CERTIFICATE_STRING
        clientID: CLIENT_ID
        clientSecret: CLIENT_SECRET
        deployCloudConsoleProxy: PROXY_BOOLEAN
        extraParams: EXTRA_PARAMS
        groupsClaim: GROUPS_CLAIM
        groupPrefix: GROUP_PREFIX
        issuerURI: ISSUER_URI
        kubectlRedirectURI: http://localhost:PORT/callback
        scopes: SCOPES
        userClaim: USER_CLAIM
        userPrefix: USER_PREFIX
    - name: NAME
      proxy: PROXY_URL
      azureAD:
        clientID: CLIENT_ID
        clientSecret: CLIENT_SECRET
        tenant: TENANT_UUID
        kubectlRedirectURI: http://localhost:PORT/callback

If you have configured more than one identity provider, you can list multiple authentication configurations in the auth-config.yaml file under the authentication anchor in the same format as in the preceding configuration. The following table describes the fields of the ClientConfig oidc and azuread object. Most of the fields are optional. The fields you need to add depend on your identity provider and the setup options chosen by your platform administrator when configuring the provider for GKE Identity Service.

Field	Required	Description	Format
name	yes	The name you want to use to identify this configuration, typically the identity provider name. A configuration name must start with a letter followed by up to 39 lowercase letters, numbers, or hyphens, and cannot end with a hyphen.	String
certificateAuthorityData	No	If provided by your platform administrator, a PEM-encoded certificate string for the identity provider. Include the resulting string in `certificateAuthorityData` as a single line.	String
clientID	Yes	The client identifier returned when registering GKE Identity Service with your provider.	String
clientSecret	Yes	The client secret returned when registering GKE Identity Service with your provider.	String
deployCloudConsoleProxy	No	Specifies whether a proxy is deployed that lets the Google Cloud console connect to an on-premises identity provider that is not publicly accessible over the internet. By default this is set to `false`.	Boolean
extraParams	No	Additional key=value parameters to send to the identity provider, specified as a comma-separated list; for example `prompt=consent,access_type=offline`.	Comma-delimited list
enableAccessToken	No	If enabled, GKE Identity Service can use the identity provider's userinfo endpoint to get groups information when a user logs in from the command line. This lets you use security groups for authorization if you have a provider (such as Okta) that provides group claims from this endpoint. If not set, this is considered to be `false`.	Boolean
groupsClaim	No	The JWT claim (field name) that your provider uses to return an account's security groups.	String
groupPrefix	No	The prefix you want to prepend to security group names to avoid clashes with existing names in your access control rules if you have configurations for multiple identity providers (typically the provider name).	String
issuerURI	Yes	The URI where authorization requests are made to your identity provider. The URI must use HTTPS.	URL String
kubectlRedirectURI	Yes	The redirect URL and port used by the gcloud CLI and specified by your platform administrator at registration, typically in the form `http://localhost:PORT/callback`.	URL String
scopes	Yes	Additional scopes to send to the OpenID provider. For example, Microsoft Azure and Okta require the `offline_access` scope.	Comma-delimited list
userClaim	No	The JWT claim (field name) that your provider uses to identify a user account. If you don't specify a value here, GKE Identity Service uses "sub", which is the user ID claim used by many providers. You can choose other claims, such as "email" or "name", depending on the OpenID provider. Claims other than "email" are prefixed with the issuer URL to prevent naming clashes.	String
userPrefix	No	The prefix you want prepended to user claims to prevent clashes with existing names, if you don't want to use the default prefix.	String
tenant	Yes	The kind of Azure AD account to be authenticated. Supported values are the tenant ID, or the tenant name for accounts belonging to a specific tenant. The tenant name is also known as the primary domain. For details on how to find these values, see Find the Microsoft Azure AD tenant ID and primary domain name.	String
proxy	No	Proxy server address to use to connect to the identity provider, if applicable. You might need to set this if, for example, your cluster is in a private network and needs to connect to a public identity provider. For example: `http://user:password@10.10.10.10:8888`.	String

SAML

The following file shows a SAML configuration:

   apiVersion: authentication.gke.io/v2alpha1
   kind: ClientConfig
   metadata:
     name: default
     namespace: kube-public
   spec:
     authentication:
     - name: NAME
       saml:
         idpEntityID: ENTITY_ID
         idpSingleSignOnURI: SIGN_ON_URI
         idpCertificateDataList: IDP_CA_CERT
         userAttribute: USER_ATTRIBUTE
         groupsAttribute: GROUPS_ATTRIBUTE
         userPrefix: USER_PREFIX
         groupPrefix: GROUP_PREFIX
         attributeMapping:
           ATTRIBUTE_KEY_1 : ATTRIBUTE_CEL_EXPRESSION_1
           ATTRIBUTE_KEY_2 : ATTRIBUTE_CEL_EXPRESSION_2
      certificateAuthorityData: CERTIFICATE_STRING
      preferredAuthentication: PREFERRED_AUTHENTICATION
      server: <>

The following table describes the fields of the ClientConfig saml object. The fields you need to add depend on your identity provider and the setup options chosen by your platform administrator when configuring the provider for GKE Identity Service.

Field	Required	Description	Format
name	yes	The name you want to use to identify this configuration, typically the identity provider name. A configuration name must start with a letter followed by up to 39 lowercase letters, numbers, or hyphens, and cannot end with a hyphen.	String
idpEntityID	Yes	The SAML entity ID for the SAML provider, specified in a URI format. For example: `https://www.idp.com/saml`.	URL String
idpSingleSignOnURI	Yes	The SAML provider's SSO endpoint, specified in a URI format. For example: `https://www.idp.com/saml/sso`.	URL String
idpCertificateDataList	Yes	Corresponds to the identity provider certificates used to verify the SAML response. These certificates must be standard base64 encoded and PEM formatted. Only a maximum of two certificates are supported to facilitate identity provider certificate rotation.	String
userAttribute	No	Name of the attribute in the SAML response that holds the username.	String
groupsAttribute	No	Name of the attribute in the SAML response that holds the user's group information.	String
userPrefix	No	The prefix you want prepended to user claims to prevent clashes with existing names, if you don't want to use the default prefix.	String
groupPrefix	No	The prefix you want to prepend to security group names to avoid clashes with existing names in your access control rules if you have configurations for multiple identity providers (typically the provider name).	String
attributeMapping	No	The mapping of additional user attributes.	String
certificateAuthorityData	No	If provided by your platform administrator, a PEM-encoded certificate string for the identity provider. Include the resulting string in `certificateAuthorityData` as a single line.	String
preferredAuthentication	No	Name of the preferred authentication method configured in the cluster.	String

LDAP

The following file shows an ldap configuration.

  apiVersion: authentication.gke.io/v2alpha1
  kind: ClientConfig
  metadata:
    name: default
    namespace: kube-public
  spec:
    authentication:
    - name: ldap
      ldap:
        server:
          host: HOST_NAME
          connectionType: CONNECTION_TYPE
          certificateAuthorityData: CERTIFICATE_AUTHORITY_DATA
        user:
          baseDn: BASE_DN
          loginAttribute: LOGIN_ATTRIBUTE
          filter: FILTER
          identifierAttribute: IDENTIFIER_ATTRIBUTE
        group:
          baseDn: BASE_DN
          filter: FILTER
          identifierAttribute: IDENTIFIER_ATTRIBUTE
        serviceAccount:
          simpleBindCredentials:
            dn: DISTINGUISHED_NAME
            password: PASSWORD

The following table describes the fields in the ClientConfig ldap object. The fields you need to add depend on your identity provider and the setup options chosen by your platform administrator when configuring the provider for GKE Identity Service:

Field	Required	Description	Format
name	yes	A name to identify this LDAP configuration	String
server
host	yes	Hostname or IP address of the LDAP server. Port is optional and will default to 389, if unspecified. For example, `ldap.server.example.com` or `10.10.10.10:389`.	String
connectionType	yes	LDAP connection type to use when connecting to the LDAP server. If `starttls` or `ldaps` is specified, the certificateAuthorityData field shouldn't be empty.	String
certificateAuthorityData	Required for certain LDAP connection types	Contains a Base64 encoded, PEM formatted certificate authority certificate for the LDAP server. This must be provided only for `ldaps` and `startTLS` connections.	String
user
baseDN	yes	The location of the subtree in the LDAP directory to search for user entries.	String in DN format.
loginAttribute	no	The name of the attribute that matches against the input username. This is used to find the user in the LDAP database, for example `(<LoginAttribute>=<username>)` and is combined with the optional filter field. This defaults to `userPrincipalName`.	String
filter	no	Optional filter to apply when searching for the user. This can be used to further restrict the user accounts that are allowed to login. If unspecified, defaults to `(objectClass=User)`.	String
identifierAttribute	no	Determines which attribute to use as the user's identity after they are authenticated. This is distinct from the `loginAttribute` field to allow users to login with a username, but then have their actual identifier be an email address or full Distinguished Name (DN). For example, setting `loginAttribute` to `sAMAccountName` and identifierAttribute to `userPrincipalName` would allow a user to login as `bsmith`, but actual RBAC policies for the user would be written as `bsmith@example.com`. Using `userPrincipalName` is recommended since this will be unique for each user. If unspecified, this defaults to `userPrincipalName`.	String
group (Optional field)
baseDN	yes	The location of the subtree in the LDAP directory to search for group entries.	String
filter	no	Optional filter to be used when searching for groups a user belongs to. This can be used to explicitly match only certain groups in order to reduce the amount of groups returned for each user. This defaults to `(objectClass=Group)`.	String
identifierAttribute	no	The identifying name of each group a user belongs to. For example, if this is set to `distinguishedName` then RBACs and other group expectations should be written as full DNs. If unspecified, this defaults to `distinguishedName`.	String
serviceAccount/simpleBindCredentials
dn	yes	The distinguished name of the service account user.	String
password	yes	The password of the service account user.	String

Enable GKE Identity Service

To enable GKE Identity Service for your project, run the following command:

gcloud container fleet identity-service enable

This creates a new GKE Identity Service controller instance to manage the lifecycle of GKE Identity Service in your fleet's clusters. You only need to run this command once per project to use GKE Identity Service with all supported clusters registered to your project fleet.

Optionally, you can enable GKE Identity Service with a fleet-level default configuration. Using this setup, the GKE Identity Service provider configuration you specify gets automatically applied to every GKE cluster on Google Cloud registered to your fleet during cluster creation. You can find out more about to do this in Configure fleet-level defaults.

Apply the configuration to a cluster

To install GKE Identity Service if necessary (EKS clusters only, all other supported cluster types already have GKE Identity Service installed by default) and apply the configuration to a cluster, run the following command:

gcloud container fleet identity-service apply \
--membership=CLUSTER_NAME \
--config=/path/to/auth-config.yaml

Replace CLUSTER_NAME with your cluster's unique name within the fleet. Upon running this command, the configuration is managed by the GKE Identity Service controller. Any local changes made to the GKE Identity Service client configuration are reconciled back by the controller to the configuration specified in this setup.

This lets GKE Identity Service retrieve Google Groups information for user accounts logging in with their Google ID. This configuration is applicable to clusters on Google Distributed Cloud (both VMware and bare metal) from GKE Enterprise version 1.13 onwards. To know more about the Google Groups feature, see Set up the connect gateway with Google Groups.

Be aware that if you have an existing configuration on your cluster for any authentication options, the following apply:

If you have existing cluster-level configurations for OIDC providers, applying a fleet-level GKE Identity Service configuration to the cluster overwrites all your existing authentication specifications.
If you have existing cluster-level configurations for providers that are not supported for fleet-level configuration, this setup will fail. You must remove the existing provider configuration to apply the fleet-level configuration.

If you no longer want the GKE Identity Service controller to manage your configuration, for example if you want to use a different authentication option or options, you can disable this feature by following the instructions in Disabling GKE Identity Service management.

Provider-specific configurations

This section provides configuration guidance for OIDC providers (such as Azure AD and Okta), including an example configuration that you can copy and edit with your own details.

Azure AD

This is the default configuration to set up GKE Identity Service with Azure AD. Using this configuration lets GKE Identity Service get user and group information from Azure AD, and lets you set up Kubernetes role based access control (RBAC) based on groups. However, using this configuration limits you to retrieving approximately 200 groups per user.

If you need to retrieve more than 200 groups per user, see the instructions for Azure AD (Advanced).

...
spec:
  authentication:
  - name: oidc-azuread
    oidc:
      clientID: CLIENT_ID
      clientSecret: CLIENT_SECRET
      cloudConsoleRedirectURI: https://console.cloud.google.com/kubernetes/oidc
      extraParams: prompt=consent, access_type=offline
      issuerURI: https://login.microsoftonline.com/TENANT_ID/v2.0
      kubectlRedirectURI: http://localhost:PORT/callback
      scopes: openid,email,offline_access
      userClaim: email

# Rest of the resource is managed by Google. DO NOT MODIFY.
...

Azure AD (Advanced)

This optional configuration for Azure AD lets GKE Identity Service retrieve user and group information with no limit on the number of groups per user, using the Microsoft Graph API. For information on platforms that support this configuration, see Advanced setup for Azure AD.

If you need to retrieve fewer than 200 groups per user, we recommend that you use the default configuration using an oidc anchor in your ClientConfig. For more information, see the instructions for Azure AD.

All fields in the example configuration are required.

...
spec:
  authentication:
  - name: azure
    azureAD:
      clientID: CLIENT_ID
      clientSecret: CLIENT_SECRET
      tenant: TENANT_UUID
      kubectlRedirectURI: http://localhost:PORT/callback
      groupFormat: GROUP_FORMAT
      userClaim: USER_CLAIM

# Rest of the resource is managed by Google. DO NOT MODIFY.
...

Replace GROUP_FORMAT with the format in which you want to retrieve group information. This field can take values corresponding to ID or NAME of the user groups. This setting is only available for clusters in Google Distributed Cloud (on-premises) deployments.

Okta

The following shows you how to set up authentication using both users and groups with Okta as your identity provider. This config lets GKE Identity Service retrieve user and group claims by using an access token and Okta's userinfo endpoint.

...
spec:
  authentication:
  - name: okta
    oidc:
      clientID: CLIENT_ID
      clientSecret: CLIENT_SECRET
      cloudConsoleRedirectURI: https://console.cloud.google.com/kubernetes/oidc
      enableAccessToken: true
      extraParams: prompt=consent
      groupsClaim: groups
      issuerURI: https://OKTA_ISSUER_URI/
      kubectlRedirectURI: http://localhost:PORT/callback
      scopes: offline_access,email,profile,groups
      userClaim: email

# Rest of the resource is managed by Google. DO NOT MODIFY.
...

Configure fleet-level defaults

You can enable GKE Identity Service with a fleet-level default configuration. Using this setup, every new GKE on Google Cloud cluster registered during cluster creation or GKE cluster will have GKE Identity Service automatically enabled on the cluster with the configuration you specify. If you have existing fleet member clusters when you enable this feature, they are not automatically updated with fleet defaults, though you can choose to apply your default configuration to them. For more information about managing fleet-level configuration, see Manage fleet-level features.

To configure GKE Identity Service with a fleet-level default configuration, do the following:

Create a file named fleet-default.yaml and populate it as per Create the configuration file.
Enable GKE Identity Service with your fleet-level default configuration:

Note: If GKE Identity Service is already enabled in your fleet without a fleet-level default configuration, use the apply command instead.
```
gcloud container fleet identity-service enable --fleet-default-member-config=fleet-default.yaml
```
To modify the existing fleet-level default configuration, or add one if GKE Identity Service is already enabled in your fleet without this feature, run the following command:
```
gcloud container fleet identity-service apply --fleet-default-member-config=default-config.yaml
```
Existing fleet member clusters that you registered prior to setting up fleet-level default configuration don't automatically inherit the default configuration. To apply the default configuration to an existing fleet member cluster, run the following command:
```
gcloud container fleet identity-service apply --origin=fleet --membership=CLUSTER_NAME
```
To disable fleet-level defaults for GKE Identity Service, run the following command to remove the default configuration:
```
gcloud container fleet identity-service delete --fleet-default-member-config
```

Verify the identity service configuration

After you complete the fleet-level setup, you can verify if the clusters in your fleet have been successfully configured with the identity service configuration that you specified.

Console

In the Google Cloud console, go to the Feature Manager page.

Go to Feature Manager
All enabled features are listed as Enabled in their panel.
Click DETAILS in the Identity Service panel. A details panel displays the status of your registered clusters.

gcloud

Run the following command:

gcloud container fleet identity-service describe

What's next

After you have configured the clusters, continue to set up user access.