Rotate your control plane IP

Autopilot Standard

---

This page explains how to perform an IP rotation for your control plane in Google Kubernetes Engine (GKE) clusters.

Overview

You can perform an IP rotation to change the IP address that your control plane uses to serve requests from the Kubernetes API. The control plane IP address is static and doesn't change unless you perform an IP rotation.

IP rotation changes the SSL certificate and cluster certificate authority (CA), which limits the ability to connect the previous address with the new one.

IP rotation also occurs as a part of credential rotation.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Perform an IP rotation

IP rotation is a multi-step process:

1. When you initiate an IP rotation, your control plane begins serving on the new IP address in addition to the original IP address.
2. GKE recreates your node pools to use the new IP address. You can also recreate your node pools manually by performing a node pool version upgrade to the same GKE version that the nodes already run.
3. After you initiate a rotation, you must update your cluster's API clients (such as development machines using the kubectl command-line interface) to begin communicating with the control plane over the new IP address.
4. When you complete the rotation, the control plane ceases serving traffic over the previous IP address.

When you start an IP address rotation, GKE recreates your nodes for you. However, during major events like Google Cloud Next, we might pause automatic node recreations so that you don't experience disruptions. If you don't complete an IP address rotation within seven days of starting it, GKE attempts to complete the rotation for you. If any nodes in your cluster still use the previous IP address, the automatic completion fails. You should plan to manually track and complete IP address rotations after you start the rotation. Don't rely on automatic completion, which is a best-effort measure.

Initiate the rotation

1. To initiate an IP rotation, run the following command:

   gcloud container clusters update CLUSTER_NAME \
       --start-ip-rotation
   

   Replace CLUSTER_NAME with the name of the cluster.

   The output is similar to the following:

   This will start an IP Rotation on cluster CLUSTER_NAME.
   The master will be updated to serve on a new IP address in addition to
   the current IP address. Google Kubernetes Engine will then schedule recreation of all nodes
   to point to the new IP address.  If maintenance window is
   used, nodes are not recreated until a maintenance window occurs. See
   documentation on how to manually update nodes. This operation is
   long-running and will block other operations on the cluster (including
   delete) until it has run to completion.
   Do you want to continue (Y/n)?
   

   This command configures the control plane to serve on two IP addresses: the original address and a new address.

   Caution: In zonal Standard clusters, which have a single-node control plane, this command causes brief downtime for the cluster API server. Autopilot clusters and regional Standard clusters have replicated control plane nodes and don't experience downtime.

2. Confirm the rotation and leave the shell open for the operation to complete.

Recreate nodes

After reconfiguring the API server to serve on a new IP address, GKE automatically updates your nodes to use the new IP address. GKE upgrades all of your nodes to the nearest supported node version, which recreates the nodes. For more information, refer to Node pool upgrades.

By default, GKE automatically completes IP address rotations seven days after you start the operation. If an active maintenance window or exclusion in your cluster prevents GKE from recreating some nodes during this seven day period, the IP address rotation fails to complete.

• If you use maintenance exclusions or maintenance windows that could result in a failed rotation, manually upgrade your cluster to force node recreation:

  gcloud container clusters upgrade CLUSTER_NAME \
      --location=LOCATION \
      --cluster-version=VERSION
  

  Replace VERSION with the same GKE version that the cluster already uses.

  For more information, see caveats for maintenance windows.

Check the progress of node pool recreation

1. To monitor the rotation operation, run the following command:

   gcloud container operations list \
       --filter="operationType=UPGRADE_NODES AND status=RUNNING" \
       --format="value(name)"
   

   This command returns the operation ID of the node upgrade operation.

2. To poll the operation, pass the operation ID to the following command:

   gcloud container operations wait OPERATION_ID
   

Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, use these instructions to poll each operation.

Update API clients

After initiating IP rotation, you must update all API clients outside the cluster (such as kubectl on developer machines) to point to the new IP address.

To update your API clients, run the following command for each client:

gcloud container clusters get-credentials CLUSTER_NAME

Note: If you are using the Kubernetes Certificates API, you also must issue new certificates.

Update hardcoded IP addresses and firewall rules

If you hardcoded the IP address of the control plane in your environment, or if you have firewall rules that target the IP address of the control plane, update the addresses to the new IP address. If you complete the rotation without updating IP addresses in applications and in firewall rules, those resources might experience disruptions when GKE stops serving on the previous control plane IP address.

Complete the rotation

After updating API clients outside the cluster, complete the rotation to configure the control plane to serve only on the new IP address.

To complete the rotation, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --complete-ip-rotation

The output is similar to the following:

This will complete the in-progress IP Rotation on cluster CLUSTER_NAME.
The master will be updated to stop serving on the old IP address and only
serve on the new IP address. Make sure all API clients have been updated
to communicate with the new IP address (e.g. by running `gcloud container
clusters get-credentials --project PROJECT_ID --region COMPUTE_REGION
CLUSTER_NAME`). This operation is long-running and will
block other operations on the cluster (including delete) until it has
run to completion.

Caution: In zonal Standard clusters, which have a single-node control plane, this command causes brief downtime for the cluster API server. Autopilot clusters and regional Standard clusters have replicated control plane nodes and don't experience downtime.

If the IP address rotation fails to complete and returns an error message similar to the following, refer to Error 400: Node pool requires recreation:

ERROR: (gcloud.container.clusters.update) ResponseError: code=400, message=Node pool "test-pool-1" requires recreation.

What's next

• Learn about Alias IPs.
• Learn about IP masquerade agent.
• Learn about configuring authorized networks.