This page explains how to perform a credential rotation in Google Kubernetes Engine (GKE).
You can perform a credential rotation to revoke and issue new credentials
for your cluster. This rotates the cluster root Certificate Authority (CA)
private key, and all certificates and private keys signed by that CA, including
the cluster client certificate (from the
MasterAuth API field), the key and
certificate for the API server, and the kubelet client certificates. See
Cluster trust for more information on how these credentials are used within a
Google recommends that you use credential rotation regularly to reduce credential lifetime and further secure your GKE cluster.
In addition to rotating credentials, credential rotation also performs an IP rotation.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
Set up default
gcloud settings using one of the following methods:
gcloud init, if you want to be walked through setting defaults.
gcloud config, to individually set your project ID, zone, and region.
Using gcloud init
If you receive the error
One of [--zone, --region] must be supplied: Please specify
location, complete this section.
gcloud initand follow the directions:
If you are using SSH on a remote server, use the
--console-onlyflag to prevent the command from launching a browser:
gcloud init --console-only
Follow the instructions to authorize
gcloudto use your Google Cloud account.
- Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.
Using gcloud config
- Set your default project ID:
gcloud config set project PROJECT_ID
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone COMPUTE_ZONE
- If you are working with Autopilot or regional clusters, set your default compute region:
gcloud config set compute/region COMPUTE_REGION
gcloudto the latest version:
gcloud components update
Check credential lifetime
You might want to check the credential lifetime before and/or after a credential rotation. Run the following
gcloud command to check the cluster root Certificate Authority (CA)'s lifetime (validity).
gcloud container clusters describe cluster-name \ --format "value(masterAuth.clusterCaCertificate)" \ | base64 --decode \ | openssl x509 -text \ | grep Validity -A 2
The output is similar to the following:
Validity Not Before: May 18 03:50:55 2021 GMT Not After : May 17 04:50:55 2026 GMT
How credential rotation works
Credential rotation is a multi-step process that includes migrating to a new IP address:
- When you initiate a credential rotation, your cluster control plane (master) begins serving on the new IP address in addition to the original IP address. New credentials are issued to workloads and the control plane.
- After you initiate a rotation, you must update your cluster's API clients
(such as development machines using the
kubectlcommand-line interface) to begin communicating with the control plane over the new IP address.
- When you complete the rotation, the control plane ceases serving traffic over the previous IP address, and old credentials are revoked.
Performing a credential rotation
The following sections explain how to perform a credential rotation.
Initiating the rotation
To initiate a credential rotation, run the following
gcloud command, which
creates new credentials and issues these credentials to the control plane. This
command also configures the control plane to serve on two IP addresses: the
original IP and a new IP.
gcloud container clusters update cluster-name --start-credential-rotation
Once the control plane has been reconfigured, GKE automatically updates your cluster's nodes to use the new IP and credentials. Each node pool is marked as "requires recreation." GKE does not finish the credential rotation until the automatic recreation is complete.
Inspecting the rotation
To monitor the rotation operation, run the following command:
gcloud container operations list | grep "AUTO_UPGRADE_NODES.*RUNNING"
This command returns the operation ID for the update operation.
To poll the operation, pass the operation ID to the following command:
gcloud container operations wait operation-id
Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, you can use the above instructions to poll each operation.
Updating API clients
Once credential rotation has been initiated, you must update all API clients outside of
the cluster (such as
kubectl on developer machines) to use the new credential.
To update your API clients, run the following command for each client:
gcloud container clusters get-credentials cluster-name
Completing the rotation
To complete the rotation, run the following command which configures the control plane to serve only with the new credential:
gcloud container clusters update cluster-name --complete-credential-rotation
- Learn about Protecting cluster metadata.
- Learn about the Kubernetes Secret object.
- Learn about rotating your IP address.