Control plane IP rotation

This page explains how to perform an IP rotation for your control plane in Google Kubernetes Engine.

Overview

You can perform an IP rotation to change the IP address that your cluster's Kubernetes master uses to serve requests from the Kubernetes API.

IP rotation also changes the SSL certificate and cluster certificate authority, so there is no externally-visible connection between the previous address and the new one.

Note: The master IP address is static and does not change unless an IP rotation is initiated.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.
• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region operational flags.

How IP rotation works

IP rotation is a multi-step process:

• When you initiate an IP rotation, your cluster master begins serving on the new IP address in addition to the original IP address.
• After you initiate a rotation, you must update your cluster's API clients (such as development machines using the kubectl command-line interface) to begin communicating with the master over the new IP address.
• When you complete the rotation, the master ceases serving traffic over the previous IP address.

Note: If you initiate an IP rotation, but do not complete it, GKE automatically completes the rotation after seven days.

Performing an IP rotation

The following sections explain how to perform an IP rotation.

Initiating the rotation

To initiate an IP rotation, run the following command:

gcloud container clusters update [CLUSTER_NAME] --start-ip-rotation

where [CLUSTER_NAME] is the name of the cluster

This command configures the cluster master to serve on two IP addresses, its original address and a new address. This causes brief downtime for the cluster API.

The command returns the following output:

This will start an IP Rotation on cluster [CLUSTER-NAME].
The master will be updated to serve on a new IP address in addition to
the current IP address. GKE will then recreate all
nodes to point to the new IP address. This operation is
long-running and will block other operations on the cluster (including
delete) until it has run to completion.
Do you want to continue (Y/n)?

After you confirm, leave your shell open until the operation is complete.

Once the master has been reconfigured, GKE automatically updates your cluster's nodes to use the new IP address. Each node pool is marked as "requires recreation." GKE does not finish the IP rotation until the automatic recreation is complete.

Note: Performing an IP rotation causes GKE to upgrade all node pools to the closest supported node version. For more information, refer to Upgrading a Container Cluster.

Inspecting the rotation

To monitor the update operation, run the following command:

gcloud container operations list --filter="UPGRADE_NODES RUNNING"

This command returns the operation ID for the update operation.

To poll the operation, pass the operation ID to the following command:

gcloud container operations wait [OPERATION_ID]

Node pools are recreated one-by-one, and each has its own operation. If you have multiple node pools, you can use the above instructions to poll each operation.

Updating API clients

Once IP rotation has been initiated, you must update all API clients outside of the cluster (such as kubectl on developer machines) to point to the new address.

To update your API clients, run the following command for each client:

gcloud container clusters get-credentials [CLUSTER_NAME]

Note: If you are using the Kubernetes Certificates API, you also must issue new certificates.

Completing the rotation

To complete the rotation, run the following command:

gcloud container clusters update [CLUSTER_NAME] --complete-ip-rotation

The command returns the following output:

This will complete the in-progress IP Rotation on cluster [CLUSTER_NAME].
The master will be updated to stop serving on the old IP address and only
serve on the new IP address. Make sure all API clients have been updated
to communicate with the new IP address (e.g. by running `gcloud container
clusters get-credentials --project [PROJECT-ID] --zone [COMPUTE-ZONE]
[CLUSTER-NAME]`). This operation is long-running and will
block other operations on the cluster (including delete) until it has
run to completion.

This command configures the cluster master to serve only on its new IP address. This causes brief downtime for the cluster API.

What's next

• Learn about Alias IPs.
• Learn about IP masquerade agent.
• Learn about configuring authorized networks.