Version 1.13. This version is no longer supported. For more information see the version support policy.

Upgrade a user cluster using Anthos On-Prem API clients

This page explains how to use the Google Cloud console or the Google Cloud CLI to upgrade an GKE on VMware user cluster that is managed by the Anthos On-Prem API. Before upgrading a user cluster, we recommend that you review Upgrade best practices.

What is the Anthos On-Prem API?

The Anthos On-Prem API is a Google Cloud-hosted API that lets you manage the lifecycle of your on-premises clusters using Terraform and standard Google Cloud tools. The Anthos On-Prem API runs in Google Cloud's infrastructure. Terraform, the Google Cloud console, and the Google Cloud CLI are clients of the API, and they use the API to create, update, upgrade, and delete clusters in your data center. If you created the cluster using a standard client, the cluster is enrolled in the Anthos On-Prem API, which means you can use the standard clients to manage the lifecycle of the cluster (with some exceptions). If you created the cluster using gkectl, you can enroll the cluster in the Anthos On-Prem API, which lets you use the standard clients.

Before you begin

Set up the gcloud CLI

To use the gcloud CLI or Terraform to upgrade a cluster:

Ensure that you have the latest version of the gcloud CLI. Update the gcloud CLI components, if needed:
```
gcloud components update
```

IAM requirements

If you aren't a project owner, you must be granted the Identity and Access Management role roles/gkeonprem.admin on the Google Cloud project that the cluster was created in. For details on the permissions included in this role, see GKE on-prem roles in the IAM documentation.

To use the console to upgrade the cluster, at a minimum, you need the following:

roles/container.viewer. This role lets users view the GKE Clusters page and other container resources in the console. For details about the permissions included in this role, or to grant a role with read/write permissions, see Kubernetes Engine roles in the IAM documentation.
roles/gkehub.viewer. This role lets users view clusters in the console. For details about the permissions included in this role, or to grant a role with read/write permissions,see GKE Hub roles in the IAM documentation.

Upgrade options

If you have a 1.13 admin cluster, you have the following options for upgrading user clusters managed by the Anthos On-Prem API:

If your admin cluster is at version 1.13.1+, you can use the upgrade feature described in the following section. This upgrade procedure is in Preview, and we recommend that you don't use this feature to upgrade clusters in a production environment.
If your admin cluster is at version 1.13.0 or lower, or for production environments, see Upgrading GKE on VMware.

Deploy the upgrade platform controller

Before upgrading a user cluster using the console or the gcloud CLI, you must deploy the upgrade platform controller on your admin cluster.

If you are unsure whether the controller has been deployed, run the following command on your admin workstation:
```
kubectl  --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    get onpremadmincluster -n kube-system -o yaml
```
Replace the following:
- ADMIN_CLUSTER_KUBECONFIG: the path of your admin cluster's kubeconfig file.
If the controller has been deployed, the following line is in the annotations section of the YAML output:
```
features.onprem.cluster.gke.io/preview-user-cluster-central-upgrade: enabled
```
If you don't see that annotation, run the command in the next step.
Run the following command on your admin workstation to deploy the upgrade platform controller:
```
gkectl update admin --enable-preview-user-cluster-central-upgrade \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    --config ADMIN_CLUSTER_CONFIG_FILE \
    --yes
```
Replace the following:
- ADMIN_CLUSTER_KUBECONFIG: the path of your admin cluster's kubeconfig file.
- ADMIN_CLUSTER_CONFIG_FILE: the path of your admin cluster's configuration file.
- Optional: The --yes flag automatically answers yes at a prompt to continue the deployment.
This command deploys a controller on the admin cluster that lets you use the Anthos On-Prem API clients to upgrade user clusters. During the user cluster upgrade, the admin cluster is enrolled with the Anthos On-Prem API if it isn't already enrolled, which eliminates the need to run any commands on the admin workstation to upgrade user clusters. When you initiate a user cluster upgrade, the Anthos On-Prem API triggers the controller to prepare for the upgrade. The controller downloads and installs bundles, and deploys the new version of the components that manage the user cluster.

The command takes about 25 minutes (or more depending on your network) to complete.

Note: The upgrade platform controller is automatically deployed on 1.14 admin clusters, both new 1.14 clusters and clusters upgraded to 1.14. You only need to deploy the controller on 1.13 admin clusters.

Upgrade a user cluster

After the upgrade platform controller is deployed on the admin cluster, you can upgrade one or more user clusters managed by the admin cluster.

To upgrade a user cluster:

Console

In the console, go to the GKE Enterprise clusters page.

Go to the GKE Enterprise clusters page
Select the Google Cloud project, and then select the cluster that you want to upgrade.
In the Details panel, click More details.
In the Cluster basics section, click Upgrade.
On the Opt in to simplified upgrades dialog box, read the text for both checkboxes, select them, and click Continue.
In the Choose target version list, select the version that you want to upgrade to. The curated list includes only the latest patch releases.
Click Upgrade.

Before the cluster is upgraded, preflight checks run to validate cluster status and node health. If the preflight checks pass, the user cluster is upgraded. It takes about 30 minutes for the upgrade to complete.

To view the status of the upgrade, click Show Details on the Cluster Details tab.

gcloud CLI

Run the following command to log in with your Google account:
```
gcloud auth login
```
Update the Google Cloud CLI components:
```
gcloud components update
```
Get a list of available versions to upgrade to:
```
gcloud container vmware clusters query-version-config \
  --cluster=USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --location=REGION
```
Replace the following:
- USER_CLUSTER_NAME: The name of the user cluster.
- FLEET_HOST_PROJECT_ID: The ID of the fleet project in which that user cluster is a member. This is the project that you specified when the cluster was created. If you created the cluster using gkectl, this is the project ID in the gkeConnect.projectID field in the cluster configuration file.
- REGION: The Google Cloud region in which the Anthos On-Prem API runs and stores its metadata. If you created the cluster using an Anthos On-Prem API client, this is the region that you specified when creating the cluster. If you created the cluster using gkectl, this is the region that you specified when you enrolled the cluster for management with the Anthos On-Prem API.
Use the following command to upgrade a cluster:
```
gcloud container vmware clusters upgrade USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --location=REGION \
  --version=VERSION
```
Replace the following:
- USER_CLUSTER_NAME: The name of the user cluster to upgrade.
- FLEET_HOST_PROJECT_ID: The ID of the fleet project in which that user cluster is a member. This is the project that you specified when the cluster was created. If you created the cluster using gkectl, this is the project ID in the gkeConnect.projectID field in the cluster configuration file.
- REGION: The Google Cloud region in which the Anthos On-Prem API runs and stores its metadata. If you created the cluster using an Anthos On-Prem API client, this is the region that you selected when creating the cluster. If you created the cluster using gkectl, this is the region that you specified when you enrolled the cluster for management with the Anthos On-Prem API
- VERSION: The GKE on VMware version that you want to upgrade to. Specify a version from the output of the previous command. We recommend that you upgrade to the most recent patch version.
It takes about 30 minutes for the upgrade to complete.

While the cluster is upgrading, run the following command in another terminal window to check the status of the cluster:

gcloud container vmware clusters describe USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --location=REGION

For addition information on the fields and flags, see gcloud container vmware clusters upgrade.

Disable the upgrade platform controller

Run the following command on your admin workstation:

gkectl update admin --disable-preview-user-cluster-central-upgrade \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    --config ADMIN_CLUSTER_CONFIG_FILE \
    --yes

To confirm that the controller has been disabled, run the following command:

kubectl  --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    get onpremadmincluster -n kube-system -o yaml

If controller has been disabled, the following line isn't in the annotations section of the YAML output:

features.onprem.cluster.gke.io/preview-user-cluster-central-upgrade: enabled