Authenticating with OpenID Connect (OIDC)

Caution: Be sure to plan your installation ahead of time. Your clusters' configurations, as declared in the GKE on-prem configuration file, become immutable after you create the clusters. You can't change most aspects of your clusters after they are created, including networking and authentication. The only configurable exceptions are:

• Resizing user clusters
• Creating additional user clusters
• Upgrading clusters
• Creating and managing node pools

This page shows how to configure GKE on-prem to use an OpenID provider for authentication to user clusters. To learn how to use OIDC with AD FS, see Authenticating with OIDC and AD FS. To learn how to use OIDC with the Google OpenID provider, see Authenticating with OIDC and Google.

For an overview of the GKE on-prem authentication flow, see Authentication.

Overview

GKE on-prem supports OpenID Connect (OIDC) as one of the authentication mechanisms for interacting with a user cluster's Kubernetes API server. With OIDC, you can manage access to Kubernetes clusters by using the standard procedures in your organization for creating, enabling, and disabling employee accounts.

There are two ways an employee can use the OIDC authentication flow:

• An employee can use kubectl to initiate an OIDC flow. To make this flow automatic, GKE on-prem provides the Anthos Plugin for Kubectl, a kubectl plugin.

Note: GKE on-prem launches a browser as a part of the authentication flow.

• An employee can use Google Cloud console to initiate an OIDC authentication flow.

In this exercise, you configure both options: kubectl and Google Cloud console.

Before you begin

This topic assumes you are familiar with OAuth 2.0 and OpenID Connect. This topic assumes you are familiar with OpenID scopes and claims.

Personas

This topic refers to three personas:

• Organization administrator: This person chooses an OpenID provider and registers client applications with the provider.

• Cluster administrator: This person creates one or more user clusters and creates authentication configuration files for developers who use the clusters.

• Developer: This person runs workloads on one or more clusters and uses OIDC to authenticate.

Choosing an OpenID provider

This section is for organization administrators.

You can use any OpenID provider of your choice. For a list of certified providers, see OpenID Certification.

One possibility is to use Active Directory Federated Services (AD FS) as your OpenID provider, and use an on-premises instance of Active Directory as your employee database. For more information, see Authenticating with OIDC and AD FS.

Another possibility is to use Google as your OpenID provider.

Creating a redirect URL for the Anthos Plugin for Kubectl

This section is for organization administrators.

As part of establishing a relationship with your OpenID provider, you must specify a redirect URL that the provider can use to return ID tokens to the Anthos Plugin for Kubectl. The Anthos Plugin for Kubectl runs on each developer's local machine and listens on a port of your choice. Choose a port number greater than 1024 that is suitable for this purpose. Then, the redirect URL is:

http://localhost:[PORT]/callback

where [PORT] is your port number.

When you configure your OpenID provider, specify http://localhost:[PORT]/callback as one of your redirect URLs. How you do this depends on your provider.

Configuring a redirect URL for Google Cloud console

This section is for organization administrators.

In addition to having a redirect URL for kubectl, you need a redirect URL for Google Cloud console. The redirect URL for Google Cloud console is:

https://console.cloud.google.com/kubernetes/oidc

When you configure your OIDC provider, specify https://console.cloud.google.com/kubernetes/oidc as one of your redirect URLs. How you do this depends on your provider.

Registering your client applications with the OpenID provider

This section is for organization administrators.

Before your developers can use the Anthos Plugin for Kubectl or Google Cloud console with your OpenID provider, you need to register those two clients with the OpenID provider. Registration includes these steps:

• Learn the provider's issuer URI. This is where the Anthos Plugin for Kubectl or Google Cloud console sends authentication requests.

• Give the provider the redirect URL for the Anthos Plugin for Kubectl.

• Give the provider the redirect URL for Google Cloud console. This is https://console.cloud.google.com/kubernetes/oidc.

• Establish a single client ID. This is the ID that the provider uses to identify both the Anthos Plugin for Kubectl and Google Cloud console.

• Establish a single client secret. The Anthos Plugin for Kubectl and Google Cloud console both use this secret to authenticate to the OpenID provider.

• Establish a custom scope that the Anthos Plugin for Kubectl or Google Cloud console can use to request the user's security groups.

• Establish a custom claim name that the provider will use to return the user's security groups.

How you perform these steps depends on your OpenID provider. To learn how to perform the registration steps with AD FS, see Authenticating with OIDC and AD FS.

Populating the oidc specification in GKE on-prem configuration file

This section is for cluster administrators.

Before you create a user cluster, you generate a GKE on-prem configuration file using gkectl create-config. The configuration includes the following oidc specification. You populate oidc with values specific to your provider:

oidc:
  issuerurl:
  kubectlredirecturl:
  clientid:
  clientsecret:
  username:
  usernameprefix:
  group:
  groupprefix:
  scopes:
  extraparams:
  usehttpproxy:
  capath:

• issuerurl: Required. URL of your OpenID provider, such as "https://example.com/adfs". Client applications, like the Anthos Plugin for Kubectl and Google Cloud console, send authorization requests to this URL. The Kubernetes API server uses this URL to discover public keys for verifying tokens. Must use HTTPS.
• kubectlredirecturl: Required. The redirect URL configured previously for the Anthos Plugin for Kubectl.
• clientid: Required. ID for the client application that makes authentication requests to the OpenID provider. Both the Anthos Plugin for Kubectl and Google Cloud console use this ID.
• clientsecret: Optional. Secret for the client application. Both the Anthos Plugin for Kubectl and Google Cloud console use this secret.
• username: Optional. JWT claim to use as the username. The default is sub, which is expected to be a unique identifier of the end user. You can choose other claims, such as email or name, depending on the OpenID provider. However, claims other than email are prefixed with the issuer URL to prevent naming clashes with other plugins.
• usernameprefix: Optional. Prefix prepended to username claims to prevent clashes with existing names. If you do not provide this field, and username is a value other than email, the prefix defaults to issuerurl#. You can use the value - to disable all prefixing.
• group: Optional. JWT claim that the provider will use to return your security groups.
• groupprefix: Optional. Prefix prepended to group claims to prevent clashes with existing names. For example, given a group foobar and a prefix gid-, gid-foobar. By default, this value is empty, and there is no prefix.
• scopes: Optional. Additional scopes to send to the OpenID provider as a comma-delimited list.
  • For authentication with Microsoft Azure or Okta, pass in offline_access.

• extraparams: Optional. Additional key-value parameters to send to the OpenID provider as a comma-delimited list.
  • For a list of authentication parameters, see Authentication URI parameters
  • If you are authorizing a group, pass in resource=token-groups-claim.
  • If your authorization server prompts for consent, pass in prompt=consent.
• usehttpproxy: Optional. Specifies whether to deploy a reverse proxy in the cluster to allow Google Cloud console access to the on-premises OIDC provider for authenticating users. Value must be a string: "true" or "false". If your identity provider is not reachable over the public internet, and you wish to authenticate using Google Cloud console, then this field must be set to "true". If left blank, this field defaults to "false".
• capath: Optional. Path to the certificate for the certificate authority (CA) that issued your identity provider's web certificate. This value might not be necessary. For example, if your identity provider's certificate was issued by a well-known public CA, then you would not need to provide a value here. However, if usehttpproxy is "true", then this value must be provided, even for a well-known public CA.

Example: Authorizing users and groups

Many providers encode user-identifying properties, such as email and user IDs, in a token. However, these properties have implicit risks for authentication policies:

• User IDs can make policies difficult to read and audit.
• Emails can create both an availability risk (if a user changes their primary email) and potentially a security risk (if an email can be re-assigned).

Therefore, it's a best practice to use group policies, as a group ID can be both persistent and easier to audit.

Suppose your provider creates identity tokens that include the following fields:

{
  'iss': 'https://server.example.com'
  'sub': 'u98523-4509823'
  'groupList': ['[e-mail]', '[e-mail]']
  ...
}

Given this token format, you'd populate your configuration file's oidc specification like so:

issueruri: 'https://server.example.com'
username: 'sub'
usernameprefix: 'uid-'
group: 'groupList'
groupprefix: 'gid-'
extraparams: 'resource=token-groups-claim'
...

After you've created the user cluster, you could then use Kubernetes role-based access control (RBAC) to grant privileged access to the authenticated users. For example, you could create a ClusterRole that grants its users read-only access to the cluster's Secrets, and create a ClusterRoleBinding resource to bind the role to the authenticated group:

ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  # The resource type for which access is granted
  resources: ["secrets"]
  # The permissions granted by the ClusterRole
  verbs: ["get", "watch", "list"]

ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-secrets-admins
subjects:
  # Allows anyone in the "us-east1-cluster-admins" group to
  # read Secrets in any namespace within this cluster.
- kind: Group
  name: gid-us-east1-cluster-admins # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
  # Allows this specific user to read Secrets in any
  # namespace within this cluster
- kind: User
  name: uid-u98523-4509823
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Creating your first authentication configuration file

This section is for cluster administrators.

After you create a user cluster, create an authentication configuration file for your cluster:

gkectl create-login-config --kubeconfig [USER_CLUSTER_KUBECONFIG]

where [USER_CLUSTER_KUBECONFIG] is the path of your user cluster's kubeconfig file. When you created your user cluster, gkectl create cluster generated a kubeconfig file for the cluster.

The preceding command creates a file named kubectl-anthos-config.yaml in the current directory.

Adding additional clusters to your authentication configuration file

This section is for cluster administrators.

You can store the authentication configuration for several clusters in a single file. Then you can distribute that one file to the developers who need access to all of those clusters.

Start with an existing authentication configuration file. Then use the following command to merge that file with the configuration for an additional cluster:

gkectl create-login-config --kubeconfig [USER_CLUSTER_KUBECONFIG] \
    --merge-from [IN_AUTH_CONFIG_FILE] --output [OUT_AUTH_CONFIG_FILE]

where

• [USER_CLUSTER_KUBECONFIG] is the kubeconfig file of the additional cluster.

• [IN_AUTH_CONFIG_FILE] is the path of the existing authentication configuration file.

• [OUT_AUTH_CONFIG_FILE] is the path where you want to save the file that holds the merged configuration. This can be the same as [IN_AUTH_CONFIG_FILE].

Note: In a single authentication configuration file, the individual cluster names must be unique.

Distributing the authentication configuration file

This section is for cluster administrators.

The authentication configuration file that you distribute to your developers must be namedkubectl-anthos-config.yaml. You can distribute the authentication configuration file in a number of ways:

• Manually give the file to each developer.

• Host the file at a URL, so developers can download it.

• Push the file to each developer's machine.

Regardless of your distribution method, for a default configuration, the authentication configuration file must be placed at this location on each developer's machine:

Linux

$HOME/.config/google/anthos/kubectl-anthos-config.yaml

macOS

$HOME/Library/Preferences/google/anthos/kubectl-anthos-config.yaml

Windows

%APPDATA%\google\anthos\kubectl-anthos-config.yaml

If you don't want to use the default configuration, you can create a custom configuration.

Placing the authentication configuration file

This section is for developers.

The authentication configuration file contains the details of all clusters you can authenticate to. Do not modify the contents of this file.

If your cluster administrator has provided you with an authentication configuration file, place the file in the correct directory. If your cluster administer has already placed the configuration on your machine, you can skip this section.

Copy your authentication configuration file to its default location:

Linux

mkdir -p  $HOME/.config/google/anthos/
cp [AUTH_CONFIG_FILE] $HOME/.config/google/anthos/kubectl-anthos-config.yaml

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

macOS

mkdir -p  $HOME/Library/Preferences/google/anthos/
cp [AUTH_CONFIG_FILE] $HOME/Library/Preferences/google/anthos/kubectl-anthos-config.yaml

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

Windows

md "%APPDATA%\google\anthos"
copy [AUTH_CONFIG_FILE] "%APPDATA%\google\anthos\kubectl-anthos-config.yaml"

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

Note: If you do not want to set up a default configuration, you can manually pass the path of the authentication configuration file to the plugin commands by using the login-config flag. For example, see Using the Anthos Plugin for Kubectl to authenticate.

Getting the Anthos Plugin for Kubectl

This section is for cluster administrators.

The Anthos Plugin for Kubectl is included in your admin workstation at:

Linux

/usr/bin/kubectl_anthos/v1.0beta/linux_amd64/kubectl-anthos

macOS

/usr/bin/kubectl_anthos/v1.0beta/darwin_amd64/kubectl-anthos

Windows

/usr/bin/kubectl_anthos/v1.0beta/windows_amd64/kubectl-anthos

You can distribute the plugin to your developers, or you can have them download the plugin directly.

Downloading the Anthos Plugin for Kubectl

This section is for cluster administrators and developers.

Each developer needs to have the Anthos Plugin for Kubectl on their own machine. Developers can download the plugin individually, or the cluster administrator can download the plugin and then distribute it to the developers.

Note for developers: Ask your cluster administrator what version of the plugin you should use.

Download the Anthos Plugin for Kubectl:

Linux

gsutil cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/linux_amd64/kubectl-anthos ./
chmod +x kubectl-anthos

macOS

gsutil cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/darwin_amd64/kubectl-anthos ./
chmod +x kubectl-anthos

Windows

gsutil cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/windows_amd64/kubectl-anthos ./
rename kubectl-anthos kubectl-anthos.exe.

Installing the Anthos Plugin for Kubectl

This section is for developers.

Your cluster administrator might provide you with the Anthos Plugin for Kubectl. Or your cluster administrator might tell you to download the plugin yourself.

To install the Anthos Plugin for Kubectl, move the executable file to any location on your PATH. For Linux and macOS, the executable file must be named kubectl-anthos. For Windows, it must be named kubectl-anthos.exe. To learn more, see the Installing kubectl plugins.

Verify that the Anthos Plugin for Kubectl is installed:

kubectl plugin list
kubectl anthos version

Caution: The plugin needs to start an internet browser to complete the kubectl anthos login command. Because of this, the plugin is not supported on headless systems.

Note: Some Windows users might have to run the command as kubectl-anthos.exe version instead of kubectl anthos version.

Note: Extending kubectl with plugins is supported in kubectl v1.12 and later.

Listing available clusters

This section is for developers.

If your authentication configuration file is at the default path, enter this command to list the clusters that you can authenticate to:

kubectl anthos listclusters

If your authentication configuration file is not at the default path, enter this command to list clusters:

kubectl anthos listclusters --loginconfig [AUTH_CONFIG_FILE]

where [AUTH_CONFIG_FILE] is the path to your authentication configuration file.

Using the Anthos Plugin for Kubectl to authenticate

This section is for developers.

Log in to a user cluster:

kubectl anthos login --cluster [CLUSTER_NAME] --user [USER_NAME] \
     --loginconfig [AUTH_CONFIG_FILE] --kubeconfig [USER_CLUSTER_KUBECONFIG]

where:

• [CLUSTER_NAME] is the name of your user cluster. This name must be selected from the list provided by the kubectl anthos listclusters command.

• [USER_NAME] an optional parameter that specifies the username for the credentials stored in the kubeconfig file. The default value is [CLUSTER_NAME]-anthos-default-user.

• [AUTH_CONFIG_FILE] is the path of your authentication configuration file. If your authentication configuration file is in the default location, you can omit this parameter.

• [USER_CLUSTER_KUBECONFIG] is the path of your user cluster's kubeconfig file. If a kubeconfig file does not exist, the command generates a new kubeconfig file from the authentication configuration file, and adds the cluster details and tokens to the kubeconfig file.

kubectl anthos login launches a browser where you can enter your credentials.

The kubeconfig file provided now contains an ID token that kubectl can use to authenticate to the Kubernetes API server on the user cluster.

Note: Some Windows users might have to run the command as kubectl-anthos.exe login instead of kubectl anthos login.

The kubectl anthos login command has an optional --dry-run flag. If you include the --dry-run flag, the command prints the kubectl commands that would add tokens to the kubeconfig file, but does not actually save the tokens in the kubeconfig file.

Verify that authentication was successful by entering a kubectl command. For example:

kubectl get nodes --kubeconfig [USER_CLUSTER_KUBECONFIG]

Using OIDC with Google Cloud console

This section is for developers.

1. Verify that your cluster is configured for OIDC.

2. Verify that your cluster has been registered with Google Cloud, either automatically during cluster creation or manually.

3. Visit the Kubernetes clusters page in Google Cloud console.

   Visit the Kubernetes clusters page

4. In the list of clusters, locate your GKE On-Prem cluster, and click Login.

5. Select Authenticate with the Identity Provider configured for the cluster, and click LOGIN.

   You will be redirected to your identity provider, where you might need to log in or consent to Google Cloud console accessing your account. Then you will be redirected back to the Kubernetes clusters page in Google Cloud console.

Custom configuration

By default, the Anthos Plugin for Kubectl expects the authentication configuration file to be at a certain location. If you do not want to put the authentication configuration file at the default location, you can manually pass the path of the authentication configuration file to the plugin commands by using the --login-config flag. For example, see Using the Anthos Plugin for Kubectl to authenticate.

Troubleshooting OIDC in GKE on-prem

Invalid configuration

If Google Cloud console cannot read the OIDC configuration from your cluster, the LOGIN button will be disabled.

Invalid provider configuration

If your identity provider configuration is invalid, you will see an error screen from your identity provider after you click LOGIN. Follow the provider-specific instructions to correctly configure the provider or your cluster.

Invalid permissions

If you complete the authentication flow, but still don't see the details of the cluster, make sure you granted the correct RBAC permissions to the account that you used with OIDC. Note that this might be a different account from the one you use to access Google Cloud console.

Error: missing 'RefreshToken' field in 'OAuth2Token' in credentials struct

You might get this error if the authorization server prompts for consent, but the required authentication parameter wasn't provided. Provide the prompt=consent parameter to GKE on-prem configuration file's oidc: extraparams field, and regenerate the client authentication file with the --extra-params prompt=consent flag.