Version 1.10

Upgrading Anthos Service Mesh on-premises

Preparing for the upgrade

Before upgrading you should check that your configuration is compatible with the new version of Anthos Service Mesh. Use the istioctl binary distributed with the new Anthos Service Mesh version and run istioctl experimental precheck to inspect the Kubernetes cluster for Istio install and upgrade requirements.

If you customized the previous installation, you need the same customizations when you upgrade to a new Anthos Service Mesh version or migrate from Istio. If you customized the installation by adding the --set values flag to istioctl install, you must add those settings to an IstioOperator YAML file, referred to as an overlay file. You specify the overlay file by using the --custom_overlay option with the filename when you run the script. The script passes the overlay file to istioctl install.

The script follows the revision upgrade process (referred to as "canary" upgrades in the Istio documentation). With a revision-based upgrade, the script installs a new revision of the control plane alongside the existing control plane. When installing the new version, the script includes a revision label that identifies the new control plane.

You then migrate to the new version by setting the same revision label on your workloads and performing a rolling restart to re-inject the proxies so that they use the new Anthos Service Mesh version and configuration. With this approach, you can monitor the effect of the upgrade on a small percentage of your workloads. After testing your application, you can migrate all traffic to the new version. This approach is much safer than doing an in-place upgrade where new control plane components replace the previous version.

Setting up your environment

You need the following tools on the machine you want to install Anthos Service Mesh from. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.

  • The curl command-line tool.
  • The Cloud SDK (the gcloud command-line tool).

After installing the Cloud SDK:

  1. Authenticate with the Cloud SDK:

    gcloud auth login
    
  2. Update the components:

    gcloud components update
    
  3. Install kubectl:

    gcloud components install kubectl
    
  4. Install the required version of kpt:

       curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v0.39.2/kpt_linux_amd64 > kpt_0_39_2
       chmod +x kpt_0_39_2
       alias kpt="$(readlink -f kpt_0_39_2)"
    
  5. Switch context to your user cluster:

    kubectl config use-context CLUSTER_NAME
  6. Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

    kubectl create clusterrolebinding cluster-admin-binding \
      --clusterrole=cluster-admin \
      --user=USER_ACCOUNT

Downloading the installation file

Linux

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-linux-amd64.tar.gz
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-linux-amd64.tar.gz.1.sig
    openssl dgst -verify /dev/stdin -signature istio-1.10.2-asm.3-linux-amd64.tar.gz.1.sig istio-1.10.2-asm.3-linux-amd64.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

     tar xzf istio-1.10.2-asm.3-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named istio-1.10.2-asm.3 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.10.2-asm.3

Mac OS

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-osx.tar.gz
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-osx.tar.gz.1.sig
    openssl dgst -sha256 -verify /dev/stdin -signature istio-1.10.2-asm.3-osx.tar.gz.1.sig istio-1.10.2-asm.3-osx.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

    tar xzf istio-1.10.2-asm.3-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.10.2-asm.3 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.10.2-asm.3

Windows

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-win.zip
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.2-asm.3-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.10.2-asm.3-win.zip.1.sig istio-1.10.2-asm.3-win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

    tar xzf istio-1.10.2-asm.3-win.zip

    The command creates an installation directory in your current working directory named istio-1.10.2-asm.3 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.10.2-asm.3

Preparing resource configuration files

When you run the istioctl install command, you include the revisioned-custom-ingressgateway.yaml file on the command line. This file allows you to control when you switch to the new version of the istio-ingressgateway after upgrading. Do the following steps to download and configure this file:

  1. Change to the directory where you want to download the anthos-service-mesh package.

  2. Download the package:

    kpt pkg get \
    https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git/asm@release-1.10-asm asm
    
  3. Set the tag to the version of Anthos Service Mesh that you are installing:

    kpt cfg set asm anthos.servicemesh.tag 1.10.2-asm.3
    
  4. Set the validating webhook to use a revision label:

    kpt cfg set asm anthos.servicemesh.rev asm-1102-3
    

    When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

Upgrading Anthos Service Mesh

To install a new version of Anthos Service Mesh, we recommend that you follow the revision-based upgrade process (also referred to as a "canary upgrade"). With a revision-based upgrade, you install a new version of the control plane alongside the existing control plane. When installing the new version, you include a revision label that identifies the version of the new control plane. Each revision is a full Anthos Service Mesh control plane implementation with its own Deployment and Service.

You then migrate to the new version by setting the same revision label on your workloads to point to the new control plane and performing a rolling restart to re-inject the proxies with the new Anthos Service Mesh version. With this approach, you can monitor the effect of the upgrade on a small percentage of your workloads. After testing your application, you can migrate all traffic to the new version. This approach is much safer than doing an in-place upgrade where a new control plane replaces the previous version of the control plane.

Updating the control plane

  1. If needed, change to the istio-1.10.2-asm.3 directory. The istioctl client is version dependent. Make sure that you use the version in the istio-1.10.2-asm.3/bin directory.

  2. Run the following command to deploy the new control plane. If you want to enable a supported optional feature, include -f and the YAML filename on the following command line. See Enabling optional features for more information.

    bin/istioctl install \
      --set profile=asm-multicloud \
      -f asm/istio/options/revisioned-istio-ingressgateway.yaml \
      --set revision=asm-1102-3

The --set revision argument adds a istio.io/rev label to istiod. After running the command, you have two control plane Deployments and Services running side-by-side:

kubectl get pods -n istio-system

Deploying and redeploying workloads

  1. Get the revision label that is on istiod and the istio-ingressgateway.

    kubectl get pod -n istio-system -L istio.io/rev
    

    The output from the command is similar to the following.

    NAME                                             READY   STATUS    RESTARTS   AGE   REV
    istio-ingressgateway-65d884685d-6hrdk            1/1     Running   0          67m
    istio-ingressgateway-65d884685d-94wgz            1/1     Running   0          67m
    istio-ingressgateway-asm-182-2-8b5fc8767-gk6hb   1/1     Running   0          5s    asm-1102-3
    istio-ingressgateway-asm-182-2-8b5fc8767-hn4w2   1/1     Running   0          20s   asm-1102-3
    istiod-asm-176-1-67998f4b55-lrzpz                1/1     Running   0          68m   asm-196-1
    istiod-asm-176-1-67998f4b55-r76kr                1/1     Running   0          68m   asm-196-1
    istiod-asm-182-2-5cd96f88f6-n7tj9                1/1     Running   0          27s   asm-1102-3
    istiod-asm-182-2-5cd96f88f6-wm68b                1/1     Running   0          27s   asm-1102-3
    1. In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1102-3.

    2. Also note the value in the revision label for the old istiod version. You need this to delete the old version of istiod when you finish moving workloads to the new version. In the example output, the value of the revision label for the old version is asm-196-1.

  2. Switch the istio-ingressgateway to the new revision. In the following command, change REVISION to the value that matches the revision label of the new version.

    kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "REVISION"}]'

    Expected output: service/istio-ingressgateway patched

  3. Add the revision label to a namespace and remove the istio-injection label (if it exists). In the following command, change REVISION to the value that matches the new revision of istiod.

    kubectl label namespace NAMESPACE istio.io/rev=REVISION istio-injection- --overwrite

    If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

  4. Restart the Pods to trigger re-injection.

    kubectl rollout restart deployment -n NAMESPACE
  5. Verify that your Pods are configured to point to the new version of istiod.

    kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
  6. Test your application to verify that the workloads are working correctly.

  7. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.

  8. If you are satisfied that your application is working as expected, continue with the steps to transition to the new version of istiod. If there's an issue with your application, follow the steps to rollback.

    Complete the transition

    If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.

    1. Change to the directory where the files from the anthos-service-mesh GitHub repository are located.

    2. Configure the validating webhook to use the new control plane.

      kubectl apply -f asm/istio/istiod-service.yaml
      
    3. Delete the old istio-ingressgatewayDeployment. The command that you run depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh:

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete deploy/istio-ingressgateway -n istio-system
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, replace OLD_REVISION with the revision label for the previous version of the istio-ingressgateway.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=OLD_REVISION -n istio-system --ignore-not-found=true
      
    4. Delete the old version of istiod. The command that you use depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh.

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, make sure that OLD_REVISION matches the revision label for the previous version of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
      
    5. Remove the old version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
      

      The expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted

    Rollback

    If you encountered an issue when testing your application with the new version of istiod, follow these steps to rollback to the previous version:

    1. Switch back to the old version of the istio-ingressgateway. In the following command, replace OLD_REVISION with the old revision.

      kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "OLD_REVISION"}]'
      
    2. Relabel your namespace to enable auto-injection with the previous version of istiod. The command that you use depends on whether you used a revision label or istio-injection=enabled with the previous version.

      • If you used a revision label for auto-injection:

        kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
        
      • If you used istio-injection=enabled:

        kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
        

      Expected output:

      namespace/NAMESPACE labeled
    3. Confirm that the revision label on the namespace matches the revision label on the previous version of istiod:

      kubectl get ns NAMESPACE --show-labels
      
    4. Restart the Pods to trigger re-injection so the proxies have the previous version:

      kubectl rollout restart deployment -n NAMESPACE
      
    5. Remove the new istio-ingressgateway Deployment. Make sure that the value of REVISION in the following command is correct.

      kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=REVISION -n istio-system --ignore-not-found=true
      
    6. Remove the new version of istiod. Make sure that the value of REVISION in the following command is correct.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION -n istio-system --ignore-not-found=true
      
    7. Remove the new version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-REVISION -n istio-system
      

      Expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-REVISION" deleted
    8. If you didn't include the --disable_canonical_service flag, the script enabled the Canonical Service controller. We recommend that you leave it enabled, but if you need to disable it, see Enabling and disabling the Canonical Service controller.

What's next