Upgrading Anthos Service Mesh on premises

This guide explains how to upgrade Anthos Service Mesh from version 1.5.4+ or 1.6.4+ to version 1.6.14 on GKE on VMware. To upgrade from Anthos Service Mesh 1.4.5+, you first have to upgrade to Anthos Service Mesh 1.5. Direct upgrades from Anthos Service Mesh 1.4 to 1.6 aren't supported.

When upgrading, we recommend that you do a dual control plane upgrade (also referred to as a canary upgrade) where both the new and previous versions of the control plane are running as you test the new version with a small percentage of your workloads. This approach is safer than an in-place upgrade, where the new version of the control plane replaces the previous version. Note that the istio-ingressgateway is upgraded in place, so you should plan for some disruption on your cluster.

Redeploying the Anthos Service Mesh control plane components takes about 5 to 10 minutes to complete. Additionally, you need to inject new sidecar proxies in all of your workloads so they are updated with the current Anthos Service Mesh version. The time it takes to update the sidecar proxies depends on many factors, such as the number of pods, the number of nodes, deployment scaling settings, pod disruption budgets, and other configuration settings. A rough estimate of the time that it takes to update the sidecar proxies is 100 pods per minute.

Preparing for the upgrade

This section outlines the steps that you take to prepare to upgrade Anthos Service Mesh.

  1. Review the Supported features and this guide to become familiar with the features and the upgrade process.

  2. If you enabled optional features when you installed the previous version of Anthos Service Mesh, you need to enable the same features when you upgrade. You enable optional features by adding --set values flags or by specifying the -f flag with a YAML file when you run the istioctl install command.

  3. If you're upgrading from Anthos Service Mesh 1.5, do the following steps in case you need to rollback:

    1. Create a directory called asm-1-5.

    2. Download the 1.5 installation file to the asm-1-5 directory.

    3. Extract the contents of the file to the asm-1-5 directory.

Setting up your environment

You need the following tools on the machine you want to install Anthos Service Mesh from. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.

After installing the Google Cloud CLI:

  1. Authenticate with the Google Cloud CLI:

    gcloud auth login

  2. Update the components:

    gcloud components update

  3. Install kubectl:

    gcloud components install kubectl

  4. Install the required version of kpt:

       curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v0.39.2/kpt_linux_amd64 > kpt_0_39_2
   chmod +x kpt_0_39_2
   alias kpt="$(readlink -f kpt_0_39_2)"

  5. Switch context to your user cluster:

    kubectl config use-context CLUSTER_NAME

  6. Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

    kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=USER_ACCOUNT

Downloading the installation file

    Linux

  1. Download the Anthos Service Mesh installation file to your current working directory: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-linux-amd64.tar.gz
  2. Download the signature file and use openssl to verify the signature: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-linux-amd64.tar.gz.1.sig
openssl dgst -verify /dev/stdin -signature istio-1.6.14-asm.2-linux-amd64.tar.gz.1.sig istio-1.6.14-asm.2-linux-amd64.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

    The expected output is: Verified OK

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory: 
    tar xzf istio-1.6.14-asm.2-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named istio-1.6.14-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

    4. Mac OS

  4. Download the Anthos Service Mesh installation file to your current working directory: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-osx.tar.gz
  5. Download the signature file and use openssl to verify the signature: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.6.14-asm.2-osx.tar.gz.1.sig istio-1.6.14-asm.2-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

    The expected output is: Verified OK

  6. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory: 
    tar xzf istio-1.6.14-asm.2-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.6.14-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

    7. Windows

  7. Download the Anthos Service Mesh installation file to your current working directory: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-win.zip
  8. Download the signature file and use openssl to verify the signature: 
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.6.14-asm.2-win.zip.1.sig
openssl dgst -verify - -signature istio-1.6.14-asm.2-win.zip.1.sig istio-1.6.14-asm.2-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

    The expected output is: Verified OK

  9. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory: 
    tar xzf istio-1.6.14-asm.2-win.zip

    The command creates an installation directory in your current working directory named istio-1.6.14-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

  10. Ensure that you're in the Anthos Service Mesh installation's root directory. 
    cd istio-1.6.14-asm.2
  11. For convenience, add the tools in the /bin directory to your PATH: 
    export PATH=$PWD/bin:$PATH

Upgrading Anthos Service Mesh

To install a new version of Anthos Service Mesh, we recommend that you follow the dual control plane upgrade process (referred to as canary upgrades in the Istio documentation). With a dual control plane upgrade, you install a new version of the control plane alongside the existing control plane. When installing the new version, you include a revision label that identifies the version of the new control plane. Each revision is a full Anthos Service Mesh control plane implementation with its own Deployment and Service.

You then migrate to the new version by setting the same revision label on your workloads to point to the new control plane and performing a rolling restart to re-inject the proxies with the new Anthos Service Mesh version. With this approach, you can monitor the effect of the upgrade on a small percentage of your workloads. After testing your application, you can migrate all traffic to the new version. This approach is much safer than doing an in-place upgrade where a new control plane replaces the previous version of the control plane.

Updating the control plane

Run the following command to deploy the new control plane. If you want to enable a supported optional feature, include -f and the YAML filename on the following command line. See Enabling optional features for more information.

istioctl install \
  --set profile=asm-multicloud \
  --set revision=asm-1614-2

The --set revision argument adds a istio.io/rev label to istiod. After running the command, you have two control plane Deployments and Services running side-by-side:

kubectl get pods -n istio-system

Example output:

NAME                                        READY   STATUS    RESTARTS   AGE
istio-ingressgateway-c56675fcd-86zdn        1/1     Running   0          2m9s
istio-ingressgateway-c56675fcd-vn4nv        1/1     Running   0          2m21s
istiod-asm-1614-2-6d5cfd4b89-xztlr       1/1     Running   0          3m44s
istiod-fb7f746f4-wcntn                      1/1     Running   0          50m
promsd-579f9f9bf4-m65nc                     2/2     Running   1          50m

Redeploying workloads

Installing the new revision has no impact on the existing sidecar proxies. To upgrade these, you must configure them to point to the new control plane. This is controlled during sidecar injection based on the namespace label istio.io/rev.

  1. Update workloads to be injected with the new Anthos Service Mesh version:

    kubectl label namespace NAMESPACE istio-injection- istio.io/rev=asm-1614-2 --overwrite

    The istio-injection label must be removed because it takes precedence over the istio.io/rev label.

  2. Restart the Pods to trigger re-injection:

    kubectl rollout restart deployment -n NAMESPACE

  3. Verify that the Pods are configured to point to the istiod-asm-1614-2 control plane:

    kubectl get pods -n NAMESPACE -l istio.io/rev=asm-1614-2

  4. Test your application to verify that the workloads are working correctly.

  5. If you have workloads in other namespaces, repeat the previous steps for each namespace.

  6. If you are satisfied that your application is working as expected, skip to Complete the upgrade. Otherwise do the following steps to rollback to the previous version.

    1. Update workloads to be injected with the previous version of the control plane:

       kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite

    2. Restart the Pods to trigger re-injection so the proxies have the previous version:

       kubectl rollout restart deployment -n NAMESPACE

    3. Rollback the control plane components:

      Rollback to previous 1.6

      1. Redeploy the previous version of the istio-ingressgateway:

        kubectl -n istio-system rollout undo deploy istio-ingressgateway

      2. Remove the new control plane:

        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-asm-1614-2 -n istio-system --ignore-not-found=true

      Rollback to 1.5

      1. Change to the directory where you downloaded the 1.5 Anthos Service Mesh installation file.

      2. Reinstall the previous version of Anthos Service Mesh. In the following command, if you enabled optional features, be sure to include the applicable --set values flags or the -f flag with the YAML file name.

        bin/istioctl install \
--set profile=asm-multicloud \

Complete the upgrade

If you are satisfied that your application is working as expected, remove the old control plane to complete the upgrade:

kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true