Updating a cluster

This document shows how to perform a cluster update that does not include a change in version.

A cluster update is a change to the cluster configuration.

A cluster upgrade is a special case of a cluster update that includes a change to the control-plane version or a change to a node pool version. For information about upgrading, see Upgrading Google Distributed Cloud.

Review your firewall rules

In version 1.29 and later, server-side preflight checks are enabled by default. Server-side preflight checks require additional firewall rules. In Firewall rules for admin clusters, search for "Preflight checks" and make sure all required firewall rules are configured.

With server-side preflight checks, when you update a user cluster using gkectl, the preflight checks are run on the admin cluster instead of locally on the admin workstation. Server-side preflight checks are run on the admin cluster when you use the Google Cloud console, the Google Cloud CLI, or Terraform to update a cluster.

When you update an admin cluster, Google Distributed Cloud deploys a Kubernetes in Docker (kind) cluster to temporarily host the Kubernetes controllers needed to update the admin cluster. This transient cluster is called a bootstrap cluster. Server-side preflight checks are run on the bootstrap cluster when you update an admin cluster.

What can be updated

Some cluster features and settings can be updated, but other features and settings can't be updated after the cluster is created. To see which features can be updated, look at the reference pages for the admin cluster configuration file and the user cluster configuration file. Fields that can be updated are marked as Mutable, and fields that can't be updated are marked as Immutable.

You can also see which features and settings can be updated by running gkectl commands.

To see what can be updated in an admin cluster:

gkectl update admin --help

Example output:

Update the admin cluster. Only the following updates are supported and they can
only be updated one at a time:
- Enabling/Disabling Vsphere Resource Metrics
- Adding static IPs
- Updating vCenter CA certificate
- Registering Admin Cluster
- Enabling/Disabling Cloud Audit Logging
- Enabling/Disabling Stackdriver
- Enabling/Disabling Auto Repair
- Enabling/Disabling Auto Resize for Addon Nodes
- Enabling/Disabling GKE OnPrem API
- Updating OS Image Type
- Enabling/Disabling AntiAffinityGroups
- Update Secrets Encryption Configuration
- [Preview] Enabling/Disabling Cluster Backup
- [Preview] Update Cluster Backup configs

To see what can be updated in a user cluster:

gkectl update cluster --help

Example output:

Update a GKE On-Prem cluster. Only the following updates are supported and they
can only be updated one at a time:
- Adding static IPs
- Updating node pool
- Updating user master cpu and memory
- Enabling/Disabling Vsphere Resource Metrics
- Enabling/Disabling vSphere CSI deployment
- Enabling/Disabling Auto Repair
- Enabling/Disabling Cloud Audit Logging
- Enabling/Disabling Stackdriver
- Enabling/Disabling GKE OnPrem API
- Registering User Cluster
- Updating vCenter CA certificate
- Updating MetalLB Address Pools
- Enabling/Disabling Auto Resizing on user master
- Updating NodePoolUpdatePolicy
- Enabling/Disabling AntiAffinityGroups
- [Preview] Enabling/Disabling Node Network Policy
- Updating Secrets Encryption
- Enabling/Disabling DataplaneV2 forwardMode

Update one thing at a time

Update only one feature or setting at a time. For example, suppose you want to update master CPU and memory and you also want to disable auto repair.

Then you would do two separate updates: one for master CPU and memory, and another auto repair. Verify that the first update is successful before you do the second update.

Update procedure

You can use gkectl, the Google Cloud console, or the Google Cloud CLI to make updates to a user cluster. If you created the user cluster using Terraform, then you can use Terraform to update the cluster. In almost all cases, you must use gkectl to update admin clusters.

The following examples show how to update clusters.

gkectl

The gkectl update command takes one of the following forms:

Admin cluster example (gkectl update admin)

Suppose you want to change the value of gkeOnPremAPI.enabled from false to true in an admin cluster. First, edit the admin cluster configuration file, and set the value to true:

gkeOnPremAPI:
  enabled: true

Then update the admin cluster:

gkectl update admin --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config ADMIN_CLUSTER_CONFIG

Replace the following:

  • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file

  • ADMIN_CLUSTER_CONFIG: the path of the admin cluster configuration file

User cluster example (gkectl update cluster)

Suppose you want to change the value of stackdriver.vSphereDisableResourceMetrics from false to true in a user cluster. First, edit the user cluster configuration file, and set the value to true:

stackdriver:
  disableVsphereResourceMetrics: true

Then update the user cluster:

gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

Replace the following:

  • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file

  • USER_CLUSTER_CONFIG: the path of the user cluster configuration file

User cluster example (gkectl update credentials)

Suppose you want to change the credentials that a user cluster uses to call vCenter Server.

The user cluster configuration file has a vCenter.credentials.fileRef.path field that points to a credentials configuration file. In the credentials configuration file, update the values of username and password. For example:

items:
- name: "vcenter-creds"
  username: "new-vcenter-account"
  password: "U$icUKEW#INE"

Then update the credentials:

gkectl update credentials vsphere \
    --config USER_CLUSTER_CONFIG \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Replace the following:

  • USER_CLUSTER_CONFIG: the path of the user cluster configuration file

  • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file

Console

  1. In the console, go to the Google Kubernetes Engine clusters overview page.

    Go to GKE clusters

  2. Select the Google Cloud project, and then select the cluster that you want to update.

  3. In the Details panel, click More details.

  4. Fields that can be updated in the Google Cloud console have a pencil icon. Click one of the pencil icons. In the box that opens, change field values as desired, and click DONE.

  5. At the top of the page, in the progress bar, click Show details. Wait for the update to complete.

gcloud CLI

Run the applicable update command. Specify only the flags for the configuration that you want to modify:

Update admin users on a user cluster

Each cluster's Kubernetes API server needs to be able to authorize requests that it receives. To configure authorization, you need to configure Kubernetes role-based access control (RBAC) policies on each cluster. An admin user is a user who has been granted the cluster-admin role in the user cluster. This role grants the user full administrative access to the cluster.

Suppose Alice and Bob are the only admin users on the user cluster called cluster-1 in the us-west1 region. And suppose Alice wants to add Trent as an additional admin user. Alice could run the following command:

gcloud container vmware clusters update cluster-1 \
    --project example-project-id-12345 \
    --location us-west1 \
    --admin-users alice@example.com \
    --admin-users bob@example.com \
    --admin-users trent@example.com

The preceding command keeps Alice and Bob in the list of admin users and adds Trent to the list of admin users. Notice that Alice and Bob must be listed in the command, because the command overwrites the current list of admin users with the list specified in the command. Also notice that each --admin-users flag specifies only one user. You can't specify multiple users in a single flag.

Terraform

  1. Change the values of the applicable fields in the Terraform configuration file that you used to create the cluster or node pool. For detailed field descriptions, see the Terraform reference documentation:

  2. Update the configuration by running terraform apply.

Update admin users on a user cluster

Each cluster's Kubernetes API server needs to be able to authorize requests that it receives. To configure authorization, you need to configure Kubernetes role-based access control (RBAC) policies on each cluster. An admin user is a user who has been granted the cluster-admin role in the user cluster. This role grants the user full administrative access to the cluster.

Suppose Alice and Bob are the only admin users in a particular user cluster. And suppose Alice wants to add Trent as an additional admin user. Alice could adjust the Terraform configuration as follows and then run terraform apply:

authorization {
  admin_users {
    username = "alice@example.com"
    username = "bob@example.com"
    username = "trent@example.com"
  }
}

The preceding adjustment keeps Alice and Bob in the list of admin users and adds Trent to the list of admin users. Notice that Alice and Bob must be listed in the new configuration, because the terraform apply command overwrites the current list of admin users with the list specified in the configuration.

More information

The documents listed in this section give additional details about updating clusters.

Update cluster/admin

The following documents provide details about using gkectl update admin and gkectl update cluster to update features and settings:

Update credentials

The following documents provide details about using gkectl update credentials to update keys and certificates:

What's next

Upgrade a cluster