Upgrade Anthos Service Mesh

Preview — asmcli

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This page explains how to:

• Run asmcli to upgrade from Anthos Service Mesh or open source Istio 1.9 or a 1.10 patch release to Anthos Service Mesh 1.10.6. Upgrades from earlier versions aren't supported.

• Do a canary upgrade to migrate your workloads to the new control plane.

Note: Adding Compute Engine virtual machines to the service mesh requires specific options when you upgrade Anthos Service Mesh. For detailed steps, see Add Compute Engine virtual machines to Anthos Service Mesh.

Before you begin

Before you begin, make sure that you:

• Review the prerequisites. Currently asmcli doesn't support upgrading Anthos Service Mesh on GKE clusters that are in a multi-cluster mesh in different projects unless you initially installed Anthos Service Mesh using asmcli.
• Review the information in Plan the upgrade.
• Install the required tools
• Download asmcli
• Grant cluster admin permissions
• Validate project and cluster

Control plane customizations

If you customized the previous installation, you need the same customizations when you upgrade to a new Anthos Service Mesh version or migrate from Istio. If you customized the installation by adding the --set values flag to istioctl install, you must add those settings to an IstioOperator YAML file, referred to as an overlay file. You specify the overlay file by using the --custom_overlay option with the filename when you run the script. The script passes the overlay file to istioctl install.

Caution: Omitting the overlay file results in a successful installation. However, your service mesh has a different configuration than you intended because the omitted values are reset back to the defaults when the control plane and ingress gateway are redeployed.

Certification Authority

Changing the certificate authority (CA) during an upgrade causes downtime. During the upgrade, mTLS traffic is interrupted until all workloads are switched to using the new control plane with the new CA.

Upgrade Anthos Service Mesh

The following outlines how to upgrade Anthos Service Mesh:

1. Run asmcli install to install Anthos Service Mesh on a single cluster. See the following sections for command line examples. The examples contain both required arguments and optional arguments that you might find useful. We recommend that you always specify the output_dir argument so that you can easily locate sample gateways and tools such as istioctl. See the navigation bar on the right for a list of the examples.

2. Optionally, install or upgrade an ingress gateway.

3. To complete setting up Anthos Service Mesh, you need to enable automatic sidecar injection and deploy or redeploy workloads.

Upgrade with default features and Mesh CA

This section shows how to run asmcli to upgrade Anthos Service Mesh with the default supported features for your platform and enable Anthos Service Mesh certificate authority (Mesh CA) as the certificate authority.

GKE

Run the following command to install the new control plane with default features. Enter your values in the provided placeholders.

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca mesh_ca

• --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
• --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
• --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
• --enable_all Allows the script to:
  • Grant required IAM permissions.
  • Enable the required Google APIs.
  • Set a label on the cluster that identifies the mesh.
  • Register the cluster to the fleet if it isn't already registered.
• --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity

On-premises

1. Set the current context to your user cluster:

   kubectl config use-context CLUSTER_NAME
   

2. Run the following command to install the new control plane with default features. Enter your values in the provided placeholders.

   ./asmcli install \
     --fleet_id FLEET_PROJECT_ID \
     --kubeconfig KUBECONFIG_FILE \
     --output_dir DIR_PATH \
     --platform multicloud \
     --enable_all \
     --ca mesh_ca
   

   • --fleet_id The project ID of the fleet host project.
   • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --platform multicloud Specifies that on-premises is the platform.
   • --enable_all Allows the script to:
     • Grant required IAM permissions.
     • Enable the required Google APIs.
     • Set a label on the cluster that identifies the mesh.
     • Register the cluster to the fleet if it isn't already registered.
   • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity

Upgrade default features with Istio CA

This section shows how to run asmcli to upgrade Anthos Service Mesh with the default supported features for your platform and enable Istio CA.

GKE

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca citadel

• --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
• --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
• --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
• --enable_all Allows the script to:
  • Grant required IAM permissions.
  • Enable the required Google APIs.
  • Set a label on the cluster that identifies the mesh.
  • Register the cluster to the fleet if it isn't already registered.
• -ca citadel Use Istio CA. Changing certificate authorities during an upgrade causes downtime.

On-premises

1. Set the current context to your user cluster:

    kubectl config use-context CLUSTER_NAME
   

2. Run the following command to install Anthos Service Mesh with default features and Istio CA:

    ./asmcli install \
      --fleet_id FLEET_PROJECT_ID \
      --kubeconfig KUBECONFIG_FILE \
      --output_dir DIR_PATH \
      --platform multicloud \
      --enable_all \
      --ca citadel \
   

   • --fleet_id The project ID of the fleet host project.
   • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --platform multicloud Specifies that on-premises is the platform.
   • --enable_all Allows the script to:
     • Grant required IAM permissions.
     • Enable the required Google APIs.
     • Set a label on the cluster that identifies the mesh.
     • Register the cluster to the fleet if it isn't already registered.

   • -ca citadel Use Istio CA. Changing the certificate authority during an upgrade causes downtime.

Upgrade with optional features

An overlay file is a YAML file containing an IstioOperator custom resource (CR) that you pass to asmcli to configure the control plane. You can override the default control plane configuration and enable an optional feature by passing the YAML file to asmcli. You can layer on more overlays, and each overlay file overrides the configuration on the previous layers.

GKE

Run the following command to install the new control plane with default features. Enter your values in the provided placeholders.

./asmcli install \
  --project_id PROJECT_ID \
  --cluster_name CLUSTER_NAME \
  --cluster_location CLUSTER_LOCATION \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --enable_all \
  --ca mesh_ca \
  --custom_overlay OVERLAY_FILE

• --project_id, --cluster_name, and --cluster_location Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.
• --fleet_id The project ID of the fleet host project. If you don't include this option, asmcli uses the project that the cluster was created in when registering the cluster.
• --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
• --enable_all Allows the script to:
  • Grant required IAM permissions.
  • Enable the required Google APIs.
  • Set a label on the cluster that identifies the mesh.
  • Register the cluster to the fleet if it isn't already registered.
• --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity
• --custom_overlay Specify the name of the overlay file.

On-premises

1. Set the current context to your user cluster:

   kubectl config use-context CLUSTER_NAME
   

2. Run the following command to install the new control plane with default features. Enter your values in the provided placeholders.

   ./asmcli install \
     --fleet_id FLEET_PROJECT_ID \
     --kubeconfig KUBECONFIG_FILE \
     --output_dir DIR_PATH \
     --platform multicloud \
     --enable_all \
     --ca mesh_ca \
     --custom_overlay OVERLAY_FILE
   

   • --fleet_id The project ID of the fleet host project.
   • --kubeconfig The path to the kubeconfig file You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --output_dir Include this option to specify a directory where asmcli downloads the anthos-service-mesh package and extracts the installation file, which contains istioctl, samples, and manifests. Otherwise asmcli downloads the files to a tmp directory. You can specify either a relative path or a full path. The environment variable $PWD doesn't work here.
   • --platform multicloud Specifies that on-premises is the platform.
   • --enable_all Allows the script to:
     • Grant required IAM permissions.
     • Enable the required Google APIs.
     • Set a label on the cluster that identifies the mesh.
     • Register the cluster to the fleet if it isn't already registered.
   • --ca mesh_ca Use Mesh CA as the certificate authority. Changing certificate authorities during an upgrade causes downtime. asmcliconfigures Mesh CA to use fleet workload identity
   • --custom_overlay Specify the name of the overlay file.

Upgrade gateways

If you have gateways deployed, you will need to upgrade these as well. For a simple upgrade follow the In-place Upgrades section in the Install and upgrade gateways guide.

Warning: If you installed Anthos Service Mesh with any other method than asmcli, the default ingress gateway installed will be removed unless you pass in the --option legacy-default-ingressgateway argument.

Switch to the new control plane

1. Get the revision label that is on istiod.

   kubectl get pod -n istio-system -L istio.io/rev
   

   The output from the command is similar to the following.

   NAME                                             READY   STATUS    RESTARTS   AGE   REV
   istiod-asm-176-1-67998f4b55-lrzpz                1/1     Running   0          68m   asm-198-3
   istiod-asm-176-1-67998f4b55-r76kr                1/1     Running   0          68m   asm-198-3
   istiod-asm-182-2-5cd96f88f6-n7tj9                1/1     Running   0          27s   asm-1106-2
   istiod-asm-182-2-5cd96f88f6-wm68b                1/1     Running   0          27s   asm-1106-2

   1. In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1106-2.

   2. Also note the value in the revision label for the old istiod version. You need this to delete the old version of istiod when you finish moving workloads to the new version. In the example output, the value of the revision label for the old version is asm-198-3.

2. Add the revision label to an application namespace and remove the istio-injection label (if it exists). In the following command, change REVISION to the value that matches the new revision of istiod.

   kubectl label namespace NAMESPACE istio.io/rev=REVISION istio-injection- --overwrite

   If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

3. Restart the Pods to trigger re-injection.

   kubectl rollout restart deployment -n NAMESPACE

4. Verify that your Pods are configured to point to the new version of istiod.

   kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION

5. Test your application to verify that the workloads are working correctly.

6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.

7. If you are satisfied that your application is working as expected, continue with the steps to transition to the new version of istiod. If there's an issue with your application, follow the steps to rollback.

   Complete the transition

   If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.

   1. Change to the directory where the files from the anthos-service-mesh GitHub repository are located.

   2. Configure the validating webhook to use the new control plane.

      kubectl apply -f asm/istio/istiod-service.yaml
      

   3. Delete the old version of istiod. The command that you use depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh.

      Migrate

      If you migrated from Istio, the old istio-ingressgateway doesn't have a revision label.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true
      

      Upgrade

      If you upgraded from a previous Anthos Service Mesh version, in the following command, make sure that OLD_REVISION matches the revision label for the previous version of istiod.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
      

   4. Remove the old version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
      

      The expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted

   Rollback

   If you encountered an issue when testing your application with the new version of istiod, follow these steps to rollback to the previous version:

   1. Relabel your namespace to enable auto-injection with the previous version of istiod. The command that you use depends on whether you used a revision label or istio-injection=enabled with the previous version.

      • If you used a revision label for auto-injection:

        kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
        

      • If you used istio-injection=enabled:

        kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
        

      Expected output:

      namespace/NAMESPACE labeled

   2. Confirm that the revision label on the namespace matches the revision label on the previous version of istiod:

      kubectl get ns NAMESPACE --show-labels
      

   3. Restart the Pods to trigger re-injection so the proxies have the previous version:

      kubectl rollout restart deployment -n NAMESPACE
      

   4. Remove the new version of istiod. Make sure that the value of REVISION in the following command is correct.

      kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION -n istio-system --ignore-not-found=true
      

   5. Remove the new version of the IstioOperator configuration.

      kubectl delete IstioOperator installed-state-REVISION -n istio-system
      

      Expected output is similar to the following:

      istiooperator.install.istio.io "installed-state-REVISION" deleted

   6. If you didn't include the --disable_canonical_service flag, the script enabled the Canonical Service controller. We recommend that you leave it enabled, but if you need to disable it, see Enabling and disabling the Canonical Service controller.

   7. If you have gateways deployed, make sure to change the revision label on the namespace or deployment to match the previous version of istiod. Follow the same process outlined under the In-place Upgrages section in the Install and upgrade gateways guide.