Upgrade clusters

Version 1.13 

• Latest
• Version 1.16
• Earlier versions (unsupported)

When you install a new version of bmctl, you can upgrade your existing clusters that were created with an earlier version. Upgrading a cluster to the latest Google Distributed Cloud version brings added features and fixes to your cluster. It also ensures that your cluster remains supported. You can upgrade admin, hybrid, standalone, or user clusters with the bmctl upgrade cluster command.

Note: When upgrading Google Distributed Cloud from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version available to ensure you have the latest security patches. Always review the release notes before upgrading so you are aware of what has changed, including security fixes and known issues. Upgrading to a lower patch version is not supported.

Upgrade considerations

The following sections outline rules and best practices to consider before you upgrade a cluster.

Preview features

Preview features are subject to change and are provided for testing and evaluation purposes only. Do not use Preview features on your production clusters. We do not guarantee that clusters that use Preview features can be upgraded. In some cases, we explicitly block upgrades for clusters that use Preview features.

For information about breaking changes related to upgrading, see the release notes.

SELinux

If you want to enable SELinux to secure your containers, you must make sure that SELinux is enabled in Enforced mode on all your host machines. Starting with Google Distributed Cloud release 1.9.0 or later, you can enable or disable SELinux before or after cluster creation or cluster upgrades. SELinux is enabled by default on Red Hat Enterprise Linux (RHEL) and CentOS. If SELinux is disabled on your host machines or you aren't sure, see Securing your containers using SELinux for instructions on how to enable it.

Google Distributed Cloud supports SELinux in only RHEL and CentOS systems.

Version rules for upgrades

When you download and install a new version of bmctl, you can upgrade your admin, hybrid, standalone, and user clusters created or upgraded with an earlier version of bmctl. Clusters can't be downgraded to a lower version.

You can only upgrade a cluster to a version that matches the version of bmctl you are using. That is, if you are using version 1.13.10 of bmctl, you can upgrade a cluster to version 1.13.10 only.

Patch version upgrades

For a given minor version, you can upgrade to any higher patch version. That is, you can upgrade a 1.13.X version cluster to version 1.13.Y as long as Y is greater than X. For example, you can upgrade from 1.12.0 to 1.12.1 and you can upgrade from 1.12.1 to 1.12.3. We recommend that you upgrade to the latest patch version whenever possible to ensure your clusters have the latest security fixes.

Minor version upgrades

You can upgrade clusters from one minor version to the next, regardless of the patch version. That is, you can upgrade from 1.N.X to 1.N+1.Y, where 1.N.X is the version of your cluster and N+1 is the next available minor version. The patch versions, X and Y, don't affect the upgrade logic in this case. For example, you can upgrade from 1.12.3 to 1.13.10.

You can't skip minor versions when upgrading clusters. If you attempt to upgrade to a minor version that is two or more minor versions higher than the cluster version, bmctl emits an error. For example, you can't upgrade a version 1.11.0 cluster to version 1.13.0.

An admin cluster can manage user clusters that are on the same or previous minor version. Managed user clusters can't be more than one minor version lower than the admin cluster, so before upgrading an admin cluster to a new minor version, make sure that all managed user clusters are at the same minor version as the admin cluster.

The examples in the following upgrade instructions show the upgrade process from version 1.12.2 to Google Distributed Cloud 1.13.10.

In-place upgrades for self-managed clusters

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Starting with Google Distributed Cloud release 1.13.1, you can perform in-place upgrades on admin, hybrid, and standalone clusters. An in-place upgrade eliminates the need for a bootstrap cluster, which simplifies the process and reduces resource requirements for an upgrade. Before you can perform an in-place upgrade on your self-managed cluster, it must be at version 1.13.0 or higher.

To perform an in-place upgrade, you can use either bmctl or kubectl:

bmctl

The upgrade process is identical to the standard upgrade process except for the bmctl upgrade cluster command.

• To start the in-place upgrade, use the --use-bootstrap=false flag with the upgrade command:

  bmctl upgrade cluster -c CLUSTER_NAME --use-bootstrap=false \
      --kubeconfig ADMIN_KUBECONFIG
  

  Replace the following:

  • CLUSTER_NAME: the name of the cluster to upgrade.
  • ADMIN_KUBECONFIG: the path to the admin cluster kubeconfig file.

As with the standard upgrade process, preflight checks are run as part of the cluster upgrade to validate cluster status and node health. If the preflight checks fail, the cluster upgrade is halted. To troubleshoot any failures, examine the cluster and related logs, since no bootstrap cluster is created.

kubectl

To upgrade a self-managed cluster with kubectl, perform the following steps:

1. Edit the cluster configuration file to set anthosBareMetalVersion to the upgrade target version.

2. To initiate the upgrade, run the following command:

   kubectl apply -f CLUSTER_CONFIG_PATH
   

   Replace CLUSTER_CONFIG_PATH with the path to the cluster configuration file.

As with the standard upgrade process, preflight checks are run as part of the cluster upgrade to validate cluster status and node health. If the preflight checks fail, the cluster upgrade is halted. To troubleshoot any failures, examine the cluster and related logs, since no bootstrap cluster is created.

Note: Although you don't need the latest version of bmctl to upgrade with kubectl, we recommend that you download the latest bmctl. You need bmctl to perform other tasks, such as health checks and backups, to ensure that your cluster stays in good working order.

Pod density

Google Distributed Cloud supports the configuration of up to 250 maximum pods per node with nodeConfig.PodDensity.MaxPodsPerNode. You can configure pod density during cluster creation only. You can't update pod density settings for existing clusters.

Known issues

For information about potential problems related to cluster upgrades, see Upgrading Anthos clusters on bare metal on the Known Issues page.

Upgrade admin, standalone, hybrid, or user clusters

When you download and install a new version of bmctl, you can upgrade your admin, hybrid, standalone, and user clusters created with an earlier version. For a given version of bmctl, a cluster can be upgraded to the same version only.

First, download the latest bmctl, then modify the appropriate cluster config files, and then issue the bmctl upgrade cluster command to complete the upgrade.

Note: You should upgrade an admin cluster before upgrading any associated user clusters. However, before upgrading the admin cluster to 1.13.10, make sure its associated user clusters are at version 1.12.0 or above. bmctl will emit an error if it finds any user cluster is below 1.12.0.

1. Download the latest bmctl from the Cloud Storage bucket and use chmod to give bmctl execute permissions to all users:

   gcloud storage cp gs://anthos-baremetal-release/bmctl/1.13.10/linux-amd64/bmctl bmctl
   chmod a+x bmctl
   

2. Modify the cluster config file to change the Google Distributed Cloud cluster version from 1.12.2 to 1.13.10. The following shows an example from an admin cluster config:

   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     name: cluster1
     namespace: cluster-cluster1
   spec:
     # Cluster type. This can be:
     #   1) admin:  to create an admin cluster. This can later be used to create user clusters.
     #   2) user:   to create a user cluster. Requires an existing admin cluster.
     #   3) hybrid: to create a hybrid cluster that runs admin cluster components and user workloads.
     #   4) standalone: to create a cluster that manages itself, runs user workloads, but does not manage other clusters.
     type: admin
     # Anthos cluster version.
     # Change the following line from 1.12.2 to 1.13.10, shown below
     anthosBareMetalVersion: 1.13.10
   

3. When upgrading clusters to 1.13.10, you must register the clusters with Connect to your project fleet, if they have not been registered already.

   1. Manually create service accounts and retrieve the JSON key files as described in Configuring service accounts for use with Connect on the Enabling Google services and service accounts page.
   2. Reference the downloaded JSON keys in the associated gkeConnectAgentServiceAccountKeyPath and gkeConnectRegisterServiceAccountKeyPath fields of the cluster config file.

4. Use the bmctl upgrade cluster command to complete the upgrade:

   bmctl upgrade cluster -c CLUSTER_NAME --kubeconfig ADMIN_KUBECONFIG
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster to upgrade.
   • ADMIN_KUBECONFIG: the path to the admin cluster kubeconfig file.

   Preflight checks are run as part of the cluster upgrade to validate cluster status and node health. The cluster upgrade doesn't proceed if the preflight checks fail.