Rotate your control plane IP

This page explains how to perform an IP rotation for your control plane in Google Kubernetes Engine (GKE) clusters running version 1.16.10-gke.8 and later.

Overview

You can perform an IP rotation to change the IP address that your control plane uses to serve requests from the Kubernetes API. The control plane IP address is static and doesn't change unless you perform an IP rotation.

IP rotation changes the SSL certificate and cluster certificate authority (CA), which limits the ability to connect the previous address with the new one.

IP rotation also occurs as a part of credential rotation.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Google Cloud CLI.
  • Set up default Google Cloud CLI settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud CLI to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Perform an IP rotation

IP rotation is a multi-step process:

  1. When you initiate an IP rotation, your control plane begins serving on the new IP address in addition to the original IP address.
  2. After you initiate a rotation, you must update your cluster's API clients (such as development machines using the kubectl command-line interface) to begin communicating with the control plane over the new IP address.
  3. When you complete the rotation, the control plane ceases serving traffic over the previous IP address.

Initiate the rotation

  1. To initiate an IP rotation, run the following command:

    gcloud container clusters update CLUSTER_NAME \
        --start-ip-rotation
    

    Replace CLUSTER_NAME with the name of the cluster.

    The output is similar to the following:

    This will start an IP Rotation on cluster CLUSTER_NAME.
    The master will be updated to serve on a new IP address in addition to
    the current IP address. Google Kubernetes Engine will then schedule recreation of all nodes
    to point to the new IP address.  If maintenance window is
    used, nodes are not recreated until a maintenance window occurs. See
    documentation on how to manually update nodes. This operation is
    long-running and will block other operations on the cluster (including
    delete) until it has run to completion.
    Do you want to continue (Y/n)?
    

    This command configures the control plane to serve on two IP addresses: the original address and a new address.

  2. Confirm the rotation and leave the shell open for the operation to complete.

After the control plane has been reconfigured, GKE schedules an update for your cluster's nodes to use the new IP address. Each node pool is marked for recreation. GKE does not finish the IP rotation until the automatic recreation is complete.

If you use maintenance windows, your nodes are not recreated until a maintenance window occurs. You can manually recreate all nodes by running the gcloud container clusters upgrade command and passing the --cluster-version flag with the same GKE version that the node pool is already running. For more information, see caveats for maintenance windows.

Inspect the rotation

  1. To monitor the node upgrade operation, run the following command:

    gcloud container operations list --filter="UPGRADE_NODES RUNNING"
    

    This command returns the operation ID of the node upgrade operation.

  2. To poll the operation, pass the operation ID to the following command:

    gcloud container operations wait OPERATION_ID
    

Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, you can use the above instructions to poll each operation.

Update API clients

After initiating IP rotation, you must update all API clients outside the cluster (such as kubectl on developer machines) to point to the new IP address.

To update your API clients, run the following command for each client:

gcloud container clusters get-credentials CLUSTER_NAME

Complete the rotation

After updating API clients outside the cluster, complete the rotation to configure the control plane to serve only on the new IP address.

To complete the rotation, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --complete-ip-rotation

The output is similar to the following:

This will complete the in-progress IP Rotation on cluster CLUSTER_NAME.
The master will be updated to stop serving on the old IP address and only
serve on the new IP address. Make sure all API clients have been updated
to communicate with the new IP address (e.g. by running `gcloud container
clusters get-credentials --project PROJECT_ID --region COMPUTE_REGION
CLUSTER_NAME`). This operation is long-running and will
block other operations on the cluster (including delete) until it has
run to completion.

What's next