Downgrading Anthos Service Mesh on premises

This guide explains how to downgrade Anthos Service Mesh from 1.6.14 to 1.5.10 on Anthos clusters on VMware.

Caution: Downgrading is a disruptive action on your cluster, and you must plan accordingly.

Redeploying the Anthos Service Mesh control plane components takes about 5 to 10 minutes to complete. Additionally, you need to inject new sidecar proxies in all of your workloads so they are updated with the current Anthos Service Mesh version. The time it takes to update the sidecar proxies depends on many factors, such as the number of pods, the number of nodes, deployment scaling settings, pod disruption budgets, and other configuration settings. A rough estimate of the time that it takes to update the sidecar proxies is 100 pods per minute.

Overview of the downgrade

This section outlines the steps that you take to downgrade Anthos Service Mesh.

Prepare

1. Review the Supported features and this guide to become familiar with the features and the downgrade process.

2. If you enabled optional features when you installed the previous version of Anthos Service Mesh, you need to enable the same features when you downgrade. You enable optional features by adding --set values flags or by specifying the -f flag with a YAML file when you run the istioctl install command.

   Caution: Omitting --set values flags or the -f flag with the YAML file results in a successful installation. However, it has a different configuration than you intended because the omitted values are reset back to their defaults when Anthos Service Mesh is redeployed. You need to use the same flags when you run istioctl install to downgrade to 1.6.14.

Downgrade

1. Follow the steps in this guide to prepare for downgrading Anthos Service Mesh.

2. Downgrade Anthos Service Mesh.

3. Update sidecar proxies

4. Test your application to verify that the workloads are working correctly.

Setting up your environment

You need the following tools on the machine you want to install Anthos Service Mesh from. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.

• The curl command-line tool.
• The Google Cloud CLI.

After installing the Google Cloud CLI:

1. Authenticate with the Google Cloud CLI:

   gcloud auth login
   

2. Update the components:

   gcloud components update
   

3. Install kubectl:

   gcloud components install kubectl
   

4. Install the required version of kpt:

      curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v0.39.2/kpt_linux_amd64 > kpt_0_39_2
      chmod +x kpt_0_39_2
      alias kpt="$(readlink -f kpt_0_39_2)"
   

   Note: Don't use gcloud to install and update kpt. Download the required kpt version.

5. Switch context to your user cluster:

   kubectl config use-context CLUSTER_NAME

6. Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

   kubectl create clusterrolebinding cluster-admin-binding \
     --clusterrole=cluster-admin \
     --user=USER_ACCOUNT

Downloading the installation file

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-linux.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-linux.tar.gz.1.sig
openssl dgst -verify /dev/stdin -signature istio-1.5.10-asm.2-linux.tar.gz.1.sig istio-1.5.10-asm.2-linux.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-linux.tar.gz

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.5.10-asm.2-osx.tar.gz.1.sig istio-1.5.10-asm.2-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-win.zip.1.sig
openssl dgst -verify - -signature istio-1.5.10-asm.2-win.zip.1.sig istio-1.5.10-asm.2-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-win.zip

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

1. Ensure that you're in the Anthos Service Mesh installation's root directory.

   cd istio-1.5.10-asm.2

2. For convenience, add the tools in the /bin directory to your PATH:

   export PATH=$PWD/bin:$PATH

Downgrading Anthos Service Mesh

To downgrade Anthos Service Mesh:

istioctl install --set profile=asm-multicloud

Check the control plane components

Downgrading requires reinstalling the control plane components, which takes about 5 to 10 minutes to complete. The old control plane components are terminated and then deleted as the new components are installed. You can check the progress by looking at the value in the AGE column of the workloads.

kubectl get pod -n istio-system

Example output:

NAME                                     READY   STATUS        RESTARTS   AGE
istio-ingressgateway-5bfdf7c586-v6wxx    2/2     Terminating   0          25m
istio-ingressgateway-7b598c5557-b88md    2/2     Running       0          5m44s
istiod-78cdbbbdb-d7tps                   1/1     Running       0          5m16s
promsd-576b8db4d6-lqf64                  2/2     Running       1          5m26s

In this example, there are two instances of istio-ingressgateway. The instance with 25min the AGE column is being terminated. All the other components are newly installed.

Updating sidecar proxies

Any workloads that were running on your cluster before the downgrade need to have the sidecar proxy re-injected so they have the current Anthos Service Mesh version.

Caution: Unless you have a load balancer or router setup for blue-green deployments, make sure you test restarting pods in a staging environment to verify that your services can handle any potential traffic interruption.

With automatic sidecar injection, you can update the sidecars for existing Pods with a Pod restart. How you restart pods depends on if they were created as part of a Deployment.

1. If you used a Deployment, restart the Deployment, which restarts all Pods with sidecars:

   kubectl rollout restart deployment -n YOUR_NAMESPACE

   If you didn't use a Deployment, delete the Pods, and they are automatically recreated with sidecars:

   kubectl delete pod -n YOUR_NAMESPACE --all

2. Check that all the Pods in the namespace have sidecars injected:

   kubectl get pod -n YOUR_NAMESPACE

   In the following example output from the previous command, notice that the READY column indicates there are two containers for each of your workloads: the primary container and the container for the sidecar proxy.

   NAME                    READY   STATUS    RESTARTS   AGE
   YOUR_WORKLOAD           2/2     Running   0          20s
   ...