Rotate your cluster credentials

Autopilot Standard

---

This page explains how to perform a credential rotation in Google Kubernetes Engine (GKE) clusters.

Overview

You can perform a credential rotation to revoke and issue new credentials for your cluster. This rotates the cluster root Certificate Authority (CA) private key, and all certificates and private keys signed by that CA, including the cluster client certificate (from the MasterAuth API field), the key and certificate for the API server, and the kubelet client certificates. For more information on how these credentials are used by your cluster, refer to Cluster trust.

We recommend that you rotate your credentials regularly to reduce credential lifetime and further secure your GKE cluster.

In addition to rotating credentials, credential rotation also performs an IP rotation.

Important: Credential rotation is not automatic. You must start credential rotation for your cluster before your current credentials expire. If your current credentials expire, your cluster enters the DEGRADED state, which prevents you from performing any cluster operations except credential rotations. The cluster remains in this state until you perform a credential rotation.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Check credential lifetime

We recommend that you check your credential lifetime before and after you perform a credential rotation so that you know the validity of your cluster root CA.

To check your credential lifetime, run the following command:

gcloud container clusters describe CLUSTER_NAME \
    --region REGION_NAME \
    --format "value(masterAuth.clusterCaCertificate)" \
    | base64 --decode \
    | openssl x509 -noout -dates

The output is similar to the following:

notBefore=Mar 17 16:45:34 2023 GMT
notAfter=Mar  9 17:45:34 2053 GMT

Perform a credential rotation

Credential rotation involves the following steps:

1. Start the rotation: the control plane starts serving on a new IP address in addition to the original IP address. New credentials are issued to workloads and the control plane.
2. Update API clients: after starting the rotation, update any cluster API clients, such as development machines using kubectl, to communicate with the control plane using the new IP address.
3. Complete the rotation: the control plane stops serving traffic over the original IP address. Old credentials are revoked.

Caution: If you start a credential rotation but do not complete it, GKE automatically completes the rotation after seven days. During this period, if a different operation requires a control plane recreation, your cluster might enter a broken state. To avoid this issue, ensure that you manually complete any credential rotations that you start.

Start the rotation

To start a credential rotation, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --region REGION_NAME \
    --start-credential-rotation

This command creates new credentials, issues these credentials to the control plane, and configures the control plane to serve on two IP addresses: the original IP and a new IP.

Caution: This command causes brief downtime for the cluster API server.

After the control plane has been reconfigured, GKE automatically updates your cluster's nodes to use the new IP and credentials. Each node pool is marked for recreation. GKE does not finish the credential rotation until the automatic recreation is complete.

Note: Performing a credential rotation causes GKE to upgrade all node pools to the closest supported node version. For more information, refer to Node pool upgrades.

If you use maintenance windows, your nodes are not recreated until a maintenance window occurs. You can manually recreate all nodes by running the gcloud container clusters upgrade command and passing the --cluster-version flag with the same GKE version that the node pool is already running. For more information, see caveats for maintenance windows.

Inspect the rotation

1. To monitor the rotation operation, run the following command:

   gcloud container operations list \
       --filter="operationType=UPGRADE_NODES AND status=RUNNING" \
       --format="value(name)"
   

   This command returns the operation ID of the node upgrade operation.

2. To poll the operation, pass the operation ID to the following command:

   gcloud container operations wait OPERATION_ID
   

Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, you can use the above instructions to poll each operation.

Update API clients

After starting the credential rotation, you must update all API clients outside the cluster (such as kubectl on developer machines) to use the new credentials and point to the new IP address of the control plane.

To update your API clients, run the following command for each client:

gcloud container clusters get-credentials CLUSTER_NAME \
    --region REGION_NAME 

Note: If you are using the Kubernetes Certificates API, you also must issue new certificates.

Complete the rotation

After updating API clients outside the cluster, complete the rotation to configure the control plane to serve only with the new credentials and the new IP address:

gcloud container clusters update CLUSTER_NAME \
    --region=REGION_NAME \
    --complete-credential-rotation

Caution: This task might cause a brief downtime for the cluster's API server, especially in zonal clusters.

If the credential rotation fails to complete and returns an error message similar to the following, refer to troubleshooting:

ERROR: (gcloud.container.clusters.update) ResponseError: code=400, message=Node pool "test-pool-1" requires recreation.

What's next

• Learn about Protecting cluster metadata.
• Learn about the Kubernetes Secret object.
• Learn about rotating your IP address.