Downgrading Anthos Service Mesh on premises

This guide explains how to downgrade Anthos Service Mesh from 1.5.10 to 1.4.10 on GKE on VMware.

Caution: Downgrading is a disruptive action on your cluster, and you must plan accordingly.

Redeploying the Anthos Service Mesh control plane components takes about 5 to 10 minutes to complete. Additionally, you need to inject new sidecar proxies in all of your workloads so they are updated with the current Anthos Service Mesh version. The time it takes to update the sidecar proxies depends on many factors, such as the number of pods, the number of nodes, deployment scaling settings, pod disruption budgets, and other configuration settings. A rough estimate of the time that it takes to update the sidecar proxies is 100 pods per minute.

Overview of the downgrade

This section outlines the steps that you take to downgrade Anthos Service Mesh.

1. Review the Supported features and this guide to become familiar with the features and the downgrade process.

2. If you enabled optional features when you installed the previous version of Anthos Service Mesh, you need to enable the same features when you downgrade. You enable optional features by adding --set values flags or by specifying the -f flag with a YAML file when you run the istioctl apply command.

   Caution: Omitting --set values flags or the -f flag with the YAML file results in a successful installation. However, it has a different configuration than you intended because the omitted values are reset back to their defaults when Anthos Service Mesh is redeployed. You need to use the same flags when you run istioctl apply to downgrade to 1.5.10.

   If you are downgrading from Anthos Service Mesh 1.5.10 to 1.4.10 and you enabled optional features in a YAML file, you need to convert the YAML from the IstioOperator API to the IstioControlPlane API.

3. Schedule a downtime. Downgrading can take up to 1 hour, depending on the scale of the cluster. Note that this doesn't include the time that you need to redeploy workloads to update sidecar proxies.

Setting up your environment

On your local machine, install and initialize the Google Cloud CLI.

Note: Updating and removing components using gcloud components is disabled if you install the gcloud CLI using apt-get or yum. To manage the gcloud CLI in this case, continue using the package management tool that you used during installation to update or remove components.

If you already have the gcloud CLI installed:

1. Authenticate with the gcloud CLI:

   gcloud auth login
   

2. Update the components:

   gcloud components update
   

3. Install kubectl:

   gcloud components install kubectl
   

4. Get the project ID of the project that the cluster was created in:

   gcloud

   gcloud projects list

   Console

   1. In the Google Cloud console, go to the Dashboard page:

      Go to the Dashboard page

   2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select your project. The project ID is displayed on the project Dashboard Project info card.

5. Set the default project ID for the Google Cloud CLI:

   gcloud config set project PROJECT_ID

Setting credentials and permissions

1. Ensure that you have kubectl for the GKE on VMware user cluster where you want to install Anthos Service Mesh. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.
2. Grant cluster admin permissions to the current user. You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

   kubectl create clusterrolebinding cluster-admin-binding \
     --clusterrole=cluster-admin \
     --user="$(gcloud config get-value core/account)"

   If you see the "cluster-admin-binding" already exists error, you can safely ignore it and continue with the existing cluster-admin-binding.

Downloading the installation file

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-linux.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-linux.tar.gz.1.sig
openssl dgst -verify - -signature istio-1.4.10-asm.18-linux.tar.gz.1.sig istio-1.4.10-asm.18-linux.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.4.10-asm.18-linux.tar.gz

The command creates an installation directory in your current working directory named istio-1.4.10-asm.18 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.4.10-asm.18-osx.tar.gz.1.sig istio-1.4.10-asm.18-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.4.10-asm.18-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.4.10-asm.18 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.10-asm.18-win.zip.1.sig
openssl dgst -verify - -signature istio-1.4.10-asm.18-win.zip.1.sig istio-1.4.10-asm.18-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.4.10-asm.18-win.zip

The command creates an installation directory in your current working directory named istio-1.4.10-asm.18 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

1. Ensure that you're in the Anthos Service Mesh installation's root directory.

   cd istio-1.4.10-asm.18

2. For convenience, add the tools in the /bin directory to your PATH:

   export PATH=$PWD/bin:$PATH

Downgrading Anthos Service Mesh

This section explains how to downgrade Anthos Service Mesh and enable the Supported default features listed on the Supported features page. For information on enabling the Supported optional features, see Enabling optional features.

To install Anthos Service Mesh:

Choose one of the following commands to configure Anthos Service Mesh in PERMISSIVE mutual TLS (mTLS) authentication mode or STRICT mTLS mode.

PERMISSIVE mTLS

istioctl manifest apply --set profile=asm-onprem

Caution: In PERMISSIVE mode, your services can accept both plaintext traffic and mutual TLS traffic at the same time and send plaintext intra-mesh traffic by default. To secure your service mesh after the downgrade, migrate your services to mutual TLS STRICT mode, as described in Mutual TLS Migration.

STRICT mTLS

istioctl manifest apply --set profile=asm-onprem \
  --set values.global.mtls.enabled=true

Check the control plane components

Downgrading requires reinstalling the control plane components, which takes about 5 to 10 minutes to complete. The old control plane components are terminated and then deleted as the new components are installed. You can check the progress by looking at the value in the AGE column of the workloads.

kubectl get pod -n istio-system

Example output

NAME                                      READY   STATUS       RESTARTS   AGE
istio-citadel-64f6d7c7c7-jtmw7            1/1     Running      0          38s
istio-galley-6b4878d445-c4rtt             1/2     Running      0          37s
istio-ingressgateway-7866c5c88f-llp28     0/1     Running      0          37s
istio-ingressgateway-7866c5c88f-m9sck     1/1     Terminating  0          25m
istio-pilot-7f4fdcb89c-r98jl              1/2     Running      0          37s
istio-sidecar-injector-65cbd565b9-q4wm9   1/1     Running      0          37s
promsd-78dfdf7c7d-2bhr6                   2/2     Running      1          37s

In this example, there are two instances of istio-ingressgateway. The instance with 25min the AGE column is being terminated. All the other components are newly installed.

Updating sidecar proxies

Any workloads that were running on your cluster before you downgraded Anthos Service Mesh need to have the sidecar proxy injected or updated so they have the current Anthos Service Mesh version.

Caution: Unless you have a load balancer or router setup for blue-green deployments, make sure you test restarting pods in a staging environment to verify that your services can handle any potential traffic interruption.

With automatic sidecar injection, you can update the sidecars for existing pods with a pod restart. How you restart pods depends on if they were created as part of a Deployment.

1. If you used a Deployment, restart the Deployment, which restarts all Pods with sidecars:

   kubectl rollout restart YOUR_DEPLOYMENT -n YOUR_NAMESPACE

   If you didn't use a Deployment, delete the Pods, and they are automatically recreated with sidecars:

   kubectl delete pod -n YOUR_NAMESPACE --all

2. Check that all the Pods in the namespace have sidecars injected:

   kubectl get pod -n YOUR_NAMESPACE --all

   In the following example output from the previous command, notice that the READY column indicates there are two containers for each of your workloads: the primary container and the container for the sidecar proxy.

   NAME                    READY   STATUS    RESTARTS   AGE
   YOUR_WORKLOAD           2/2     Running   0          20s
   ...