Credential rotation

This page explains how to perform a credential rotation in Google Kubernetes Engine.

Overview

You can perform a Credential rotation to revoke and issue new credentials for your cluster. This rotates the cluster root Certificate Authority (CA) private key, and all certificates and private keys signed by that CA, including the cluster client certificate (from MasterAuth API field), the master API key and certificate, and the kubelet client certificates. See Cluster Trust for more information on how these credentials are used within a cluster.

Google recommends that you use credential rotation regularly to reduce credential lifetime and further secure your GKE cluster. These are not automatically rotated.

In addition to rotating credentials, Credential rotation also performs an IP rotation.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone.

Using gcloud config

  • Set your default project ID:
    gcloud config set project project-id
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone compute-zone
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region compute-region
  • Update gcloud to the latest version:
    gcloud components update

How credential rotation works

Credential rotation is a multi-step process that includes migrating to a new IP address:

  • When you initiate a credential rotation, your cluster master begins serving on the new IP address in addition to the original IP address. New credentials are issued to workloads and the control plane.
  • After you initiate a rotation, you must update your cluster's API clients (such as development machines using the kubectl command-line interface) to begin communicating with the master over the new IP address.
  • When you complete the rotation, the master ceases serving traffic over the previous IP address, and old credentials are revoked.

Performing a credential rotation

The following sections explain how to perform a credential rotation.

Initiating the rotation

To initiate a credential rotation, run the following command, which creates new credentials and issues them to the control plane. It also configures the cluster master to serve on two IP Address, the original IP and a new IP. This causes brief downtime for the cluster API.

gcloud container clusters update cluster-name --start-credential-rotation

Once the master has been reconfigured, GKE automatically updates your cluster's nodes to use the new IP and credentials. Each node pool is marked as "requires recreation." GKE does not finish the credential rotation until the automatic recreation is complete.

Inspecting the rotation

To monitor the rotation operation, run the following command:

gcloud container operations list | grep "AUTO_UPGRADE_NODES.*RUNNING"

This command returns the operation ID for the update operation.

To poll the operation, pass the operation ID to the following command:

gcloud container operations wait operation-id

Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, you can use the above instructions to poll each operation.

Updating API clients

Once credential rotation has been initiated, you must update all API clients outside of the cluster (such as kubectl on developer machines) to use the new credential.

To update your API clients, run the following command for each client:

gcloud container clusters get-credentials cluster-name

Completing the rotation

To complete the rotation, run the following command which configures the cluster master to serve only with the new credential. This causes brief downtime for the cluster API.

gcloud container clusters update cluster-name --complete-credential-rotation

What's next