Version 1.11

Upgrade Anthos Service Mesh

This page explains how to:

  • Run asmclito upgrade from Anthos Service Mesh to Anthos Service Mesh 1.11.2.

  • Optionally, deploy an ingress gateway.

  • Do a canary upgrade to migrate your workloads to the new control plane.

The Anthos Service Mesh version that you can upgrade from differs by platform.

GKE

You can upgrade directly to Anthos Service Mesh 1.11.2-asm.17 on Google Kubernetes Engine from the following versions:

1.9 and 1.10

On-premises

You can upgrade directly to Anthos Service Mesh 1.11.2-asm.17 on Anthos clusters on VMware and Anthos on bare metal from the following versions:

1.10

Anthos clusters on AWS

You can upgrade directly to Anthos Service Mesh 1.11.2-asm.17 on Anthos clusters on AWS from the following versions:

1.10

Amazon EKS

If you have Anthos Service Mesh 1.7 installed on EKS, you will need to install Anthos Service Mesh 1.11 on a new cluster. Upgrades from Anthos Service Mesh 1.7 to Anthos Service Mesh 1.11 aren't supported.

Before you begin

Before you begin, make sure that you:

Control plane customizations

If you customized the previous installation, you need the same customizations when you upgrade to a new Anthos Service Mesh version or migrate from Istio. If you customized the installation by adding the --set values flag to istioctl install, you must add those settings to an IstioOperator YAML file, referred to as an overlay file. You specify the overlay file by using the --custom_overlay option with the filename when you run the script. The script passes the overlay file to istioctl install.

Certification Authority

Changing the certificate authority (CA) during an upgrade causes downtime. During the upgrade, mTLS traffic is interrupted until all workloads are switched to using the new control plane with the new CA.

Upgrade Anthos Service Mesh

The following outlines how to upgrade Anthos Service Mesh:

  1. If you are upgrading a multi-cluster mesh on GKE that uses Anthos Service Mesh certificate authority (Mesh CA), run asmcli create-mesh to configure the multi-cluster mesh to trust fleet workload identity for no down time cross-cluster load balancing during the upgrade.

  2. Run asmcli install to install Anthos Service Mesh on a single cluster. See the following sections for command line examples. The examples contain both required arguments and optional arguments that you might find useful. We recommend that you always specify the output_dir argument so that you can easily locate sample gateways and tools such as istioctl. See the navigation bar on the right for a list of the examples.

  3. Optionally, install or upgrade an ingress gateway. By default, asmcli doesn't install the istio-ingressgateway. We recommend that you deploy and manage the control plane and gateways separately. If you need the default istio-ingressgateway installed with the in-cluster control plane, include the --option legacy-default-ingressgateway argument.

  4. To complete setting up Anthos Service Mesh, you need to enable automatic sidecar injection and deploy or redeploy workloads.

Configure the multi-cluster mesh to trust fleet workload identity

If you are upgrading a multi-cluster mesh on GKE that uses Mesh CA as the certificate authority, you need to run asmcli create-mesh before upgrading each cluster. This command configures Mesh CA to use the fleet workload identity pool, FLEET_PROJECT_ID.svc.id.goog, as the trust domain after you upgrade. The asmcli create-mesh command:

  • Registers all clusters to the same fleet.
  • Configures the mesh to trust the fleet workload identity.
  • Creates remote secrets.

You can either specify the URI for each cluster or the path the kubeconfig file.

Cluster URI

In the following command, replace FLEET_PROJECT_ID with the project ID of the fleet host project and the cluster URI with the cluster name, zone or region, and project ID for each cluster.

./asmcli create-mesh \
    FLEET_PROJECT_ID \
    CLUSTER_NAME_1/CLUSTER_LOCATION_1/PROJECT_ID_1 \
    CLUSTER_NAME_2/CLUSTER_LOCATION_2/PROJECT_ID_2 # \
    # Add a line for each cluster in the mesh

kubeconfig file

In the following command, replace FLEET_PROJECT_ID with the project ID of the fleet host project and PATH_TO_KUBECONFIG with the path to each kubeconfig file.

./asmcli create-mesh \
    FLEET_PROJECT_ID \
    PATH_TO_KUBECONFIG_1 \
    PATH_TO_KUBECONFIG_2 # \
    # Add a line for each cluster in the mesh

Upgrade with default features and Mesh CA

This section shows how to run asmcli to upgrade Anthos Service Mesh with the default supported features for your platform and enable Anthos Service Mesh certificate authority (Mesh CA) as the certificate authority.

GKE

Run the following command to upgrade the control plane with default features and Mesh CA. Enter your values in the provided placeholders.

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca mesh_ca
  • --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
  • --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
  • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
  • --enable_all Allows the script to:
    • Grant required IAM permissions.
    • Enable the required Google APIs.
    • Set a label on the cluster that identifies the mesh.
    • Register the cluster to the fleet if it isn't already registered.
  • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity

On-premises

Run the following commands on Anthos clusters on VMware or Anthos on bare metal to upgrade the control plane with default features and Mesh CA. Enter your values in the provided placeholders.

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca mesh_ca
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity

Upgrade default features with Istio CA

This section shows how to run asmcli to upgrade Anthos Service Mesh with the default supported features for your platform and enable Istio CA.

GKE

Run the following command to upgrade the control plane with default features and Istio CA. Enter your values in the provided placeholders.

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca citadel
  • --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
  • --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
  • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
  • --enable_all Allows the script to:
    • Grant required IAM permissions.
    • Enable the required Google APIs.
    • Set a label on the cluster that identifies the mesh.
    • Register the cluster to the fleet if it isn't already registered.
  • -ca citadel Use Istio CA. Changing certificate authorities during an upgrade causes downtime.

On-premises

Run the following commands on Anthos clusters on VMware or Anthos on bare metal to upgrade the control plane with default features and Istio CA. Enter your values in the provided placeholders.

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca citadel \
      --ca_cert FILE_PATH \
      --ca_key FILE_PATH \
      --root_cert FILE_PATH \
      --cert_chain FILE_PATH
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • -ca citadel Use Istio CA as the certificate authority.
    • --ca_cert The intermediate certificate
    • --ca_key The key for the intermediate certificate
    • --root_cert The root certificate
    • --cert_chain The certificate chain

AWS

Run the following commands on Anthos clusters on AWS to upgrade the control plane with default features and Istio CA. Enter your values in the provided placeholders. You can choose to enable Ingress for the public subnet or the private subnet.

Public

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca citadel \
      --ca_cert FILE_PATH \
      --ca_key FILE_PATH \
      --root_cert FILE_PATH \
      --cert_chain FILE_PATH
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • -ca citadel Use Istio CA as the certificate authority.
    • --ca_cert The intermediate certificate
    • --ca_key The key for the intermediate certificate
    • --root_cert The root certificate
    • --cert_chain The certificate chain

Private

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Save the following YAML to a file called istio-operator-internal-lb.yaml:

    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      components:
        ingressGateways:
        - enabled: true
          k8s:
            serviceAnnotations:
              service.beta.kubernetes.io/aws-load-balancer-internal: "true"
          name: istio-ingressgateway
    
  3. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca citadel \
      --ca_cert FILE_PATH \
      --ca_key FILE_PATH \
      --root_cert FILE_PATH \
      --cert_chain FILE_PATH
      --custom_overlay istio-operator-internal-lb.yaml
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • -ca citadel Use Istio CA as the certificate authority.
    • --ca_cert The intermediate certificate
    • --ca_key The key for the intermediate certificate
    • --root_cert The root certificate
    • --cert_chain The certificate chain

Amazon EKS

Run the following commands on Amazon EKS to upgrade the control plane with default features and Istio CA. Enter your values in the provided placeholders.

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca citadel \
      --ca_cert FILE_PATH \
      --ca_key FILE_PATH \
      --root_cert FILE_PATH \
      --cert_chain FILE_PATH
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • -ca citadel Use Istio CA as the certificate authority.
    • --ca_cert The intermediate certificate
    • --ca_key The key for the intermediate certificate
    • --root_cert The root certificate
    • --cert_chain The certificate chain

Upgrade with optional features

An overlay file is a YAML file containing an IstioOperator custom resource (CR) that you pass to asmcli to configure the control plane. You can override the default control plane configuration and enable an optional feature by passing the YAML file to asmcli. You can layer on more overlays, and each overlay file overrides the configuration on the previous layers.

GKE

Run the following command to install the control plane with an optional feature. To add multiple files, specify --custom_overlay and the filename, for example: --custom_overlayoverlay_file1.yaml --custom_overlay overlay_file2.yaml --custom_overlay overlay_file3.yaml Enter your values in the provided placeholders.

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca mesh_ca \
  --custom_overlay OVERLAY_FILE
  • --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
  • --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
  • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
  • --enable_all Allows the script to:
    • Grant required IAM permissions.
    • Enable the required Google APIs.
    • Set a label on the cluster that identifies the mesh.
    • Register the cluster to the fleet if it isn't already registered.
  • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity
  • --custom_overlay Specify the name of the overlay file.

Outside Google Cloud

Run the following commands on Anthos clusters on VMware, Anthos on bare metal, Anthos clusters on AWS, or Amazon EKS. Enter your values in the provided placeholders.

  1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
    
  2. Run asmcli install:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca mesh_ca \
      --custom_overlay OVERLAY_FILE
    
    • --fleet_id The project ID of the fleet host project.
    • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
    • --platform multicloud Specifies that on-premises is the platform.
    • --enable_all Allows the script to:
      • Grant required IAM permissions.
      • Enable the required Google APIs.
      • Set a label on the cluster that identifies the mesh.
      • Register the cluster to the fleet if it isn't already registered.
    • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity
    • --custom_overlay Specify the name of the overlay file.

Upgrade gateways

If you have gateways deployed, you will need to upgrade these as well. For a simple upgrade follow the In-place Upgrades section in the Install and upgrade gateways guide.

Switch to the new control plane

  1. Get the revision label that is on istiod.

    kubectl get pod -n istio-system -L istio.io/rev
    

    The output from the command is similar to the following.

    NAME                                             READY   STATUS    RESTARTS   AGE   REV
    istiod-asm-176-1-67998f4b55-lrzpz                1/1     Running   0          68m   asm-1104-14
    istiod-asm-176-1-67998f4b55-r76kr                1/1     Running   0          68m   asm-1104-14
    istiod-asm-182-2-5cd96f88f6-n7tj9                1/1     Running   0          27s   asm-1112-17
    istiod-asm-182-2-5cd96f88f6-wm68b                1/1     Running   0          27s   asm-1112-17
    1. In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1112-17.

    2. Also note the value in the revision label for the old istiod version. You need this to delete the old version of istiod when you finish moving workloads to the new version. In the example output, the value of the revision label for the old version is asm-1104-14.

  2. Add the revision label to an application namespace and remove the istio-injection label (if it exists). In the following command, change REVISION to the value that matches the new revision of istiod.

    kubectl label namespace NAMESPACE istio.io/rev=REVISION istio-injection- --overwrite

    If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

  3. Restart the Pods to trigger re-injection.

    kubectl rollout restart deployment -n NAMESPACE
  4. Test your application to verify that the workloads are working correctly.

  5. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.

  6. If you are satisfied that your application is working as expected, continue with the steps to transition to the new version of istiod. If there's an issue with your application, follow the steps to rollback.

    Complete the transition

    If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.

    1. Change to the directory where the files from the anthos-service-mesh GitHub repository are located.

    2. Configure the validating webhook to use the new control plane.

      kubectl apply -f asm/istio/istiod-service.yaml
      
    3. Delete the old version of istiod. The command that you use depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh.

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, make sure that OLD_REVISION matches the revision label for the previous version of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
      
    4. Remove the old version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
      

      The expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted

    Rollback

    If you encountered an issue when testing your application with the new version of istiod, follow these steps to rollback to the previous version:

    1. Relabel your namespace to enable auto-injection with the previous version of istiod. The command that you use depends on whether you used a revision label or istio-injection=enabled with the previous version.

      • If you used a revision label for auto-injection:

        kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
        
      • If you used istio-injection=enabled:

        kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
        

      Expected output:

      namespace/NAMESPACE labeled
    2. Confirm that the revision label on the namespace matches the revision label on the previous version of istiod:

      kubectl get ns NAMESPACE --show-labels
      
    3. Restart the Pods to trigger re-injection so the proxies have the previous version:

      kubectl rollout restart deployment -n NAMESPACE
      
    4. Remove the new version of istiod. Make sure that the value of REVISION in the following command is correct.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION -n istio-system --ignore-not-found=true
      
    5. Remove the new version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-REVISION -n istio-system
      

      Expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-REVISION" deleted
    6. If you didn't include the --disable_canonical_service flag, asmcli enabled the Canonical Service controller. We recommend that you leave it enabled, but if you need to disable it, see Enabling and disabling the Canonical Service controller.

    7. If you have gateways deployed, make sure to change the revision label on the namespace or deployment to match the previous version of istiod. Follow the same process outlined under the In-place Upgrages section in the Install and upgrade gateways guide.