Version 1.8

Downgrading Anthos Service Mesh on GKE

This guide explains how to downgrade Anthos Service Mesh from 1.8.1 to 1.7.3 on Anthos clusters on Google Cloud.

Redeploying the Anthos Service Mesh control plane components takes about 5 to 10 minutes to complete. Additionally, you need to inject new sidecar proxies in all of your workloads so they are updated with the current Anthos Service Mesh version. The time it takes to update the sidecar proxies depends on many factors, such as the number of pods, the number of nodes, deployment scaling settings, pod disruption budgets, and other configuration settings. A rough estimate of the time that it takes to update the sidecar proxies is 100 pods per minute.

Overview of the downgrade

This section outlines the steps that you take to downgrade Anthos Service Mesh.

Prepare

  1. Review the Supported features and this guide to become familiar with the features and the downgrade process.

  2. If you enabled optional features when you installed the previous version of Anthos Service Mesh, you need to enable the same features when you downgrade. You enable optional features by adding --set values flags or by specifying the -f flag with a YAML file when you run the istioctl install command.

  3. If you are downgrading Anthos Service Mesh on a private cluster, you must add a firewall rule to open port 15017 if you want to use automatic sidecar injection. If you don't add the firewall rule and automatic sidecar injection is enabled, you get an error when you deploy workloads. For details on adding a firewall rule, see Adding firewall rules for specific use cases.

Downgrade

  1. Follow the steps in this guide to prepare for downgrading Anthos Service Mesh.

  2. Downgrade Anthos Service Mesh.

  3. Update sidecar proxies

  4. Test your application to verify that the workloads are working correctly.

Setting up your environment

For installations on Google Kubernetes Engine, you can follow the installation guides using Cloud Shell, an in-browser command line interface to your Google Cloud resources, or your own computer running Linux or macOS.

Use Cloud Shell

Cloud Shell provisions a g1-small Compute Engine virtual machine (VM) running a Debian-based Linux operating system. The advantages to using Cloud Shell are:

  • Cloud Shell includes the gcloud, kubectl and kpt command-line tools that you need.

  • Your Cloud Shell $HOME directory has 5GB persistent storage space.

  • You have your choice of text editors:

    • Code editor, which you access by clicking at the top of the Cloud Shell window.

    • Emacs, Vim, or Nano, which you access from the command line in Cloud Shell.

To use Cloud Shell:

  1. Go to the Cloud Console.
  2. Select your Cloud project.
  3. Click the Activate Cloud Shell button at the top of the Cloud Console window.

    Google Cloud Platform console

    A Cloud Shell session opens inside a new frame at the bottom of the Cloud Console and displays a command-line prompt.

    Cloud Shell session

  4. Update the components:

    gcloud components update
    

    The command responds with output similar to the following:

    ERROR: (gcloud.components.update)
    You cannot perform this action because the Cloud SDK component manager
    is disabled for this installation. You can run the following command
    to achieve the same result for this installation:
    
    sudo apt-get update && sudo apt-get --only-upgrade install ...
  5. Copy the long command and paste it to update the components.

Use Linux or macOS

On your local machine, install and initialize the Cloud SDK (the gcloud command-line tool).

If you already have the Cloud SDK installed:

  1. Authenticate with the Cloud SDK:

    gcloud auth login
    
  2. Update the components:

    gcloud components update
    
  3. Install kubectl:

    gcloud components install kubectl
    
  4. Install kpt:

    gcloud components install kpt
    
  5. Make sure that Git is in your path so that kpt can find it.

Setting environment variables

  1. Get the project ID for the project that the cluster was created in and the project number for the environ host project.

    gcloud

    Run the following command:

    gcloud projects list
    

    Console

    1. Go to the Dashboard page in the Cloud Console.

      Go to the Dashboard page

    2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select your project.

      The project ID is displayed on the project Dashboard Project info card.

  2. Create an environment variable for the project ID of the project that the cluster was created in:

    export PROJECT_ID=YOUR_PROJECT_ID

  3. Create an environment variable for the project number of the environ host project:

    export ENVIRON_PROJECT_NUMBER=YOUR_ENVIRON_PROJECT_NUMBER

  4. Create the following environment variables:

    • Set the cluster name:

      export CLUSTER_NAME=YOUR_CLUSTER_NAME
    • Set the CLUSTER_LOCATION to either your cluster zone or cluster region:

      export CLUSTER_LOCATION=YOUR_ZONE_OR_REGION

Setting credentials and permissions

  1. Get authentication credentials to interact with the cluster:

    gcloud container clusters get-credentials ${CLUSTER_NAME} \
        --project=${PROJECT_ID}
    
  2. Grant cluster admin permissions to the current user. You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

    kubectl create clusterrolebinding cluster-admin-binding \
      --clusterrole=cluster-admin \
      --user="$(gcloud config get-value core/account)"

If you see the "cluster-admin-binding" already exists error, you can safely ignore it and continue with the existing cluster-admin-binding.

Downloading the installation file

    Linux

  1. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-linux.tar.gz
  2. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-linux.tar.gz.1.sig
    openssl dgst -verify /dev/stdin -signature istio-1.7.3-asm.6-linux.tar.gz.1.sig istio-1.7.3-asm.6-linux.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.7.3-asm.6-linux.tar.gz

    The command creates an installation directory in your current working directory named istio-1.7.3-asm.6 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

  4. Mac OS

  5. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-osx.tar.gz
  6. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-osx.tar.gz.1.sig
    openssl dgst -sha256 -verify /dev/stdin -signature istio-1.7.3-asm.6-osx.tar.gz.1.sig istio-1.7.3-asm.6-osx.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  7. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.7.3-asm.6-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.7.3-asm.6 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

  8. Windows

  9. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-win.zip
  10. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.3-asm.6-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.7.3-asm.6-win.zip.1.sig istio-1.7.3-asm.6-win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  11. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.7.3-asm.6-win.zip

    The command creates an installation directory in your current working directory named istio-1.7.3-asm.6 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

  12. Ensure that you're in the Anthos Service Mesh installation's root directory.
    cd istio-1.7.3-asm.6
  13. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH

Preparing resource configuration files

When you run the istioctl install command, you specify -f istio-operator.yaml on the command line. This file contains information about your project and cluster that Anthos Service Mesh requires. You need to download a package that contains istio-operator.yaml and other resource configuration files so that you can set the project and cluster information.

To prepare the resource configuration files:

  1. Create a new directory for the Anthos Service Mesh package resource configuration files. We recommend that you use the cluster name as the directory name.

  2. Change to the directory where you want to download the Anthos Service Mesh package.

  3. Download the asm package:

    kpt pkg get \
    https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git/asm@release-1.7-asm asm
    
  4. Set the project ID for the project that the cluster was created in:

    kpt cfg set asm gcloud.core.project ${PROJECT_ID}
    
  5. Set the cluster name:

    kpt cfg set asm gcloud.container.cluster ${CLUSTER_NAME}
    
  6. Set the default zone or region:

    kpt cfg set asm gcloud.compute.location ${CLUSTER_LOCATION}
    

Downgrading Anthos Service Mesh

To downgrade Anthos Service Mesh:

istioctl install \
  -f asm/cluster/istio-operator.yaml \
  --set profile=asm-gcp

Check the control plane components

Downgrading requires reinstalling the control plane components, which takes about 5 to 10 minutes to complete. The old control plane components are terminated and then deleted as the new components are installed. You can check the progress by looking at the value in the AGE column of the workloads.

kubectl get pod -n istio-system

Example output:

NAME                                     READY   STATUS        RESTARTS   AGE
istio-ingressgateway-5bfdf7c586-v6wxx    2/2     Terminating   0          25m
istio-ingressgateway-7b598c5557-b88md    2/2     Running       0          5m44s
istiod-78cdbbbdb-d7tps                   1/1     Running       0          5m16s
promsd-576b8db4d6-lqf64                  2/2     Running       1          5m26s

In this example, there are two instances of istio-ingressgateway. The instance with 25min the AGE column is being terminated. All the other components are newly installed.

Updating sidecar proxies

Any workloads that were running on your cluster before the downgrade need to have the sidecar proxy re-injected so they have the current Anthos Service Mesh version.

With automatic sidecar injection, you can update the sidecars for existing Pods with a Pod restart. How you restart pods depends on if they were created as part of a Deployment.

  1. If you used a Deployment, restart the Deployment, which restarts all Pods with sidecars:

    kubectl rollout restart deployment -n YOUR_NAMESPACE

    If you didn't use a Deployment, delete the Pods, and they are automatically recreated with sidecars:

    kubectl delete pod -n YOUR_NAMESPACE --all
  2. Check that all the Pods in the namespace have sidecars injected:

    kubectl get pod -n YOUR_NAMESPACE

    In the following example output from the previous command, notice that the READY column indicates there are two containers for each of your workloads: the primary container and the container for the sidecar proxy.

    NAME                    READY   STATUS    RESTARTS   AGE
    YOUR_WORKLOAD           2/2     Running   0          20s
    ...