Upgrading Apigee hybrid to version 1.8

Introducing Apigee ingress gateway

Starting in version 1.8, Apigee hybrid offers a new feature to manage the ingress gateway for your hybrid installation, Apigee ingress gateway. Anthos Service Mesh is no longer a prerequisite for hybrid installation. With Apigee ingress gateway, Apigee will stop supplying routing configuration to Anthos Service Mesh. After the upgrade, you have to migrate the traffic to the new Apigee ingress gateway before you can start using the feature.

Apigee uses a small subset of Anthos Service Mesh features for the ingress gateway. Starting with hybrid version 1.8 Apigee hybrid includes an ingress gateway which is installed and upgraded as part of Apigee hybrid upgrades. Therefore you do not need to build expertise around Anthos Service Mesh to install, upgrade, and manage Apigee hybrid. Issues around ingress gateway versions and compatibility with Apigee hybrid releases are handled automatically.

Two scenarios for migrating are:

  • Multi-cluster or multi-region migration (recommended):

    Before switching to a new Ingress for Apigee, drain all the traffic to another cluster or region from the cluster you are migrating. This will give you time to test if the new Apigee ingress gateway is working as expected. Then shift the traffic back to the upgraded cluster.

  • In-place upgrade (not recommended in production environments):

    During the upgrade Apigee will bring up the new ingress gateway with an IP address you specify. You can then test if the new Apigee ingress gateway is working as expected, and then shift traffic to the new ingress. There might be downtime during this upgrade.

When upgrading Apigee hybrid to version 1.8, you must configure Apigee ingress gateway in your overrides file. After upgrading, you control which type of ingress gateway your clusters will use by directing the A or CNAME records at your registrar to the IP address for Apigee ingress gateway or for Anthos Service Mesh.

Upgrading to version 1.8.8 overview

The procedures for upgrading Apigee hybrid are organized in the following sections:

  1. Prepare to upgrade.
  2. Install hybrid runtime version 1.8.8.
  3. For the ingress gateway, pick one of the following options:

Prerequisite

These upgrade instructions assume you have Apigee hybrid version 1.7.x or an earlier patch release of version 1.8.x installed and wish to upgrade it to version 1.8.8. If you are updating from an earlier version see the instructions for Upgrading Apigee hybrid to version 1.7.

If you prefer to continue using Anthos Service Mesh, you must ensure that Anthos Service Mesh is upgraded to a supported version. See the Supported platforms table for supported versions of Anthos Service Mesh.

Prepare to upgrade to version 1.8

  1. These instructions use the environment variable APIGEECTL_HOME for the directory in your file system where you have installed apigeectl. If needed, change directory into your apigeectl directory and define the variable with the following command:

    Linux

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Mac OS

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Windows

    set APIGEECTL_HOME=%CD%
    echo %APIGEECTL_HOME%
  2. Make a backup copy of your version 1.7 $APIGEECTL_HOME/ directory. For example:
    tar -czvf $APIGEECTL_HOME/../apigeectl-v1.7-backup.tar.gz $APIGEECTL_HOME
  3. Back up your Cassandra database following the instructions in Cassandra backup and recovery

Add the Cloud Trace Agent role to the service account for the Apigee runtime. (Optional)

Optional: If you plan to use Cloud trace and you have not already performed this step on your hybrid v1.7 installation, ensure your service account for your Apigee runtime services has the Cloud Trace Agent Google role. (roles/cloudtrace.agent).

For production environments, this is usually the apigee-runtime service account. For non-production environments, this is usually the apigee-non-prod service account.

You can add the role in the Cloud console > IAM & Admin > Service accounts UI or with the following commands:

  1. Get the email address for your service account with the following command:

    Production

    gcloud iam service-accounts list --filter "apigee-runtime"

    If it matches the pattern apigee-runtime@$ORG_NAME.iam.gserviceaccount.com, you can use that pattern in the next step.

    Non-Prod

    gcloud iam service-accounts list --filter "apigee-non-prod"

    If it matches the pattern apigee-non-prod@$ORG_NAME.iam.gserviceaccount.com, you can use that pattern in the next step.

  2. Assign the Cloud Trace Agent role to the service account:

    Production

    gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/cloudtrace.agent"

    Non-Prod

    gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-non-prod@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/cloudtrace.agent"

    Example

    gcloud projects add-iam-policy-binding hybrid-example-project \
        --member="serviceAccount:apigee-runtime@hybrid-example-project.iam.gserviceaccount.com" \
        --role="roles/cloudtrace.agent"

    Where: $PROJECT_ID is the name of the Google Cloud project where Apigee hybrid is installed.

Prepare to install Apigee ingress gateway

To install Apigee ingress gateway as part of the upgrade. You need to add the following ingressGateways property to your overrides file.

Syntax

ingressGateways:
- name: INGRESS_NAME
  replicaCountMin: REPLICAS_MIN
  replicaCountMax: REPLICAS_MAX
  resources:
    requests:
      cpu: CPU_COUNT_REQ
      memory: MEMORY_REQ
    limits:
      cpu: CPU_COUNT_LIMIT
      memory: MEMORY_LIMIT
  svcAnnotations:  # optional. See Known issue 243599452.
    SVC_ANNOTATIONS_KEY: SVC_ANNOTATIONS_VALUE
  svcLoadBalancerIP: SVC_LOAD_BALANCER_IP # optional

Example

ingressGateways:
- name: prod1
  replicaCountMin: 2
  replicaCountMax: 100
  resources:
    requests:
      cpu: 1
      memory: 1Gi
    limits:
      cpu: 2
      memory: 2Gi 
  • INGRESS_NAME is the name of the ingress deployment. This can be any name that meets the following requirements:
    • Have a maximum length of 17 characters
    • Contain only lowercase alphanumeric characters, '-' or '.'
    • Start with an alphanumeric character
    • End with an alphanumeric character

    See ingressGateways[].name in the Configuration property reference

  • REPLICAS_MIN and REPLICAS_MAX are the minimum and maximum replica counts for Apigee ingress gateway in your installation. For more information and default settings, see ingressGateways[].replicaCountMin and ingressGateways[].replicaCountMax in the Configuration property reference.
  • CPU_COUNT_REQ and MEMORY_REQ are the CPU and memory request for each replica of Apigee ingress gateway in your installation.

    For more information and default settings, see ingressGateways[].resources.requests.cpu and ingressGateways[].resources.requests.memory in the Configuration property reference.

  • CPU_COUNT_LIMIT and MEMORY_LIMIT The maximum CPU and memory limits for each replica of Apigee ingress gateway in your installation.

    For more information and default settings, see ingressGateways[].resources.limits.cpu and ingressGateways[].resources.limits.memory in the Configuration property reference.

  • SVC_ANNOTATIONS_KEY and SVC_ANNOTATIONS_VALUE (optional):

    This is a key-value pair that provides annotations for your default ingress service. Annotations are used by your cloud platform to help configure your hybrid installation, for example setting the loadbalancer type to either internal or external. For example:

    ingressGateways:
      svcAnnotations:
        networking.gke.io/load-balancer-type: "Internal"

    Annotations vary from platform to platform. Refer to your platform documentation for required and suggested annotations.

    See ingressGateways[].svcAnnotations in the Configuration property reference.
  • SVC_LOAD_BALANCER_IP (optional) Allows you to assign a static IP address for your load balancer. On platforms that support specifying the load balancer IP address, the load balancer will be created with this IP address. On platforms that do not allow you to specify the load balancer IP address, this property is ignored.

    If you do not have a static IP address allocated for your load balancer, leave this property out of your overrides file.

    See ingressGateways[].svcLoadBalancerIP in the Configuration property reference.

Make additional changes to your overrides file to enable or disable optional v1.8 features

Add the following properties to your overrides.yaml file to enable new features in hybrid v1.8. These features are optional.

  • Org-scoped UDCA is now on by default. Using a single UDCA deployment to handle traffic for all environments prevents under-utilization of UDCA pods and increases availability of node resources for other Apigee components. Org-scoped UDCA uses a single service account for all environments, apigee-udca.

    If you are using different service accounts for UDCA in different environments, be aware that it will now use the service account specified at the org level in your overrides file with udca:serviceAccountPath, instead of the ones specified at the env level with envs:udca:serviceAccountPath.

    Apigee hybrid v 1.8 supports environment-scoped UDCA. To keep per-environment UDCA, set orgScopedUDCA: false.

    See orgScopedUDCA in the Configuration properties reference.

  • Enable validateOrg to require strict validation that the Apigee organization and environment are active and work with the Google Cloud Platform project specified in your overrides file.
    validateOrg: true

    See validateOrg in the Configuration properties reference.

Install the hybrid 1.8.8 runtime

  1. Be sure you are in the hybrid base directory (the parent of the directory where the apigeectl executable file is located):
    cd $APIGEECTL_HOME/..
  2. Download the release package for your operating system using the following command. Be sure to select your platform in the following table:

    Linux

    Linux 64 bit:

    curl -LO \
      https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_linux_64.tar.gz

    Mac OS

    Mac 64 bit:

    curl -LO \
      https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_mac_64.tar.gz

    Windows

    Windows 64 bit:

    curl -LO ^
      https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.8.8/apigeectl_windows_64.zip
  3. Rename your current apigeectl/ directory to a backup directory name. For example:

    Linux

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/

    Mac OS

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.7/ 

    Windows

    rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.7 
  4. Extract the downloaded gzip file contents into your hybrid base directory. The hybrid base directory is the directory where the renamed apigeectl-v1.7 directory is located:

    Linux

    tar xvzf filename.tar.gz -C ./

    Mac OS

    tar xvzf filename.tar.gz -C ./

    Windows

    tar xvzf filename.zip -C ./
  5. The tar contents are, by default, expanded into a directory with the version and platform in its name. For example: ./apigeectl_1.8.8-xxxxxxx_linux_64. Rename that directory to apigeectl using the following command:

    Linux

    mv apigeectl_1.8.8-xxxxxxx_linux_64 apigeectl

    Mac OS

    mv apigeectl_1.8.8-xxxxxxx_mac_64 apigeectl

    Windows

    rename apigeectl_1.8.8-xxxxxxx_windows_64 apigeectl
  6. Change to the apigeectl directory:
    cd ./apigeectl

    This directory is the apigeectl home directory. It is where the apigeectl executable command is located.

  7. These instructions use the environment variable $APIGEECTL_HOME for the directory in your file system where the apigeectl utility is installed. If needed, change directory into your apigeectl directory and define the variable with the following command:

    Linux

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Mac OS

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Windows

    set APIGEECTL_HOME=%CD%
    echo %APIGEECTL_HOME%
  8. Verify the version of apigeectl with the version command:
    ./apigeectl version
    Version: 1.8.8
  9. Move to the hybrid-base-directory/hybrid-files directory. The hybrid-files directory is where configuration files such as the overrides file, certs, and service accounts are located. For example:
    cd $APIGEECTL_HOME/../hybrid-files
  10. Verify that kubectl is set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid.
    kubectl config get-contexts | grep \*
  11. In the hybrid-files directory:
    1. Update the following symbolic links to $APIGEECTL_HOME. These links allow you to run the newly installed apigeectl command from inside the hybrid-files directory:
      ln -nfs $APIGEECTL_HOME/tools tools
      ln -nfs $APIGEECTL_HOME/config config
      ln -nfs $APIGEECTL_HOME/templates templates
      ln -nfs $APIGEECTL_HOME/plugins plugins
    2. To check that the symlinks were created correctly, execute the following command and make sure the link paths point to the correct locations:
      ls -l | grep ^l
  12. Do a dry run initialization to check for errors:
    ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client

    Where OVERRIDES_FILE is the name of your overrides file, for example ./overrides/overrides.yaml.

  13. If there are no errors, initialize hybrid 1.8.8. This command also installs and configures Apigee ingress gateway:
    $APIGEECTL_HOME/apigeectl init -f OVERRIDES_FILE
  14. Check the initialization status:
    $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

    On success, the output says: All containers ready.

    As a further check, you can also run this command to check ApigeeDataStore status:

    kubectl describe apigeeds -n apigee

    In the output, look for State: running.

  15. Check for errors with a dry run of the apply command using the --dry-run flag:
    $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --dry-run=client
  16. If there are no errors, apply your overrides. Select and follow the instructions for production environments or non-prod environments, depending on your installation.

    Production

    For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.

    1. Be sure you are in the hybrid-files directory.
    2. Apply your overrides to upgrade Cassandra:
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --datastore
    3. Check completion:
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

      Proceed to the next step only when the pods are ready.

    4. Apply your overrides to upgrade Telemetry components and check completion:
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --telemetry
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    5. Bring up Redis components:
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --redis
    6. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --org
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    7. Apply your overrides to upgrade your environments. You have two choices:
      • Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
        $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --env ENV_NAME
        $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

        Where ENV_NAME is the name of the environment you are upgrading.

      • All environments at one time: Apply your overrides to all environments at once and check completion:
        $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --all-envs
        $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    8. Apply your overrides to upgrade the virtualhosts components and check completion:
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --settings virtualhosts
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

    Non-prod

    In most non-production, demo, or experimental environments, you can apply the overrides to all components at once. If your non-production environment is large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments.

    1. Be sure you are in the hybrid-files directory.
    2. $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE
    3. Check the status:
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

Upgrade your Kubernetes version

Upgrade your Kubernetes platform to the versions supported by hybrid 1.8. Follow your platform's documentation if you need help.

Switch traffic from Anthos Service Mesh to Apigee ingress gateway

To switch your traffic to Apigee ingress gateway:

  1. Expose the Apigee ingress gateway. Follow the procedures in Expose Apigee ingress gateway.
  2. Test your new ingress gateway by calling a proxy. Ideally, test all crucial proxies you currently have deployed.
  3. To switch the traffic, update your DNS records to point to the IP address for your new Apigee ingress gateway. Depending on your DNS provider, you might be able to gradually shift traffic to the new endpoint. Tip: You can find the external IP address of Apigee ingress gateway with the following command:
    kubectl get svc -n apigee -l app=apigee-ingressgateway

    Your output should look something like:

    NAME                                        TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                      AGE
    apigee-ingressgateway-prod-hybrid-37a39bd   LoadBalancer   192.0.2.123   233.252.0.123   15021:32049/TCP,80:31624/TCP,443:30723/TCP   16h
  4. Ensure all the runtime traffic is working by monitoring your dashboards. Only proceed to the next step if everything is working as expected. Make sure no traffic is going through your old ingress gateway (Anthos Service Mesh), as the DNS update may be slow to propagate because of DNS caching.
  5. To stop Apigee from supplying configuration to Anthos Service Mesh, follow the steps in Stop supplying configuration to ASM in the Managing Apigee ingress gateway guide.
  6. Retest and monitor API proxy traffic.
  7. Follow the instructions in the Anthos Service Mesh documentation to Uninstall Anthos Service Mesh from the cluster.

Upgrade Anthos Service Mesh to version 1.15

Perform the procedures using the Anthos Service Mesh documentation appropriate for your platform:

The instructions to install and configure Anthos Service Mesh are different depending on your platform. The platforms are divided into the following categories:

  • GKE: Google Kubernetes Engine clusters running on Google Cloud.
  • Outside Google Cloud: Anthos clusters running on:
    • Anthos clusters on VMware (GKE on-prem)
    • Anthos on bare metal
    • Anthos clusters on AWS
    • Amazon EKS
  • Other Kubernetes Platforms: Conformant clusters created and running on:
    • AKS
    • EKS
    • OpenShift

GKE

The sequence for upgrading to Anthos Service Mesh version 1.13.9 for your hybrid installation is as follows:

  1. Prepare for the upgrade.
  2. Install the new version of Anthos Service Mesh.
  3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
  4. Upgrade your gateways and configure the new webhooks.

Prepare to upgrade Anthos Service Mesh to version 1.13.9

  1. Review the requirements in Upgrade Anthos Service Mesh, but do not perform the upgrade yet.
  2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the current istiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
    echo $DELETE_REV

    Your output should look something like 1.12.9-asm.2

  3. Create a new overlay.yaml file or verify that your existing overlay.yaml contains the following contents:
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: true
          k8s:
            nodeSelector:
              # default node selector, if different or not using node selectors, change accordingly.
              cloud.google.com/gke-nodepool: apigee-runtime
            resources:
              requests:
                cpu: 1000m
            service:
              type: LoadBalancer
              loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
              ports:
                - name: http-status-port
                  port: 15021
                - name: http2
                  port: 80
                  targetPort: 8080
                - name: https
                  port: 443
                  targetPort: 8443
      meshConfig:
        accessLogFormat:
          '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:
    1. Download asmcli
    2. Grant cluster admin permissions
    3. Validate project and cluster
    4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section".
  5. Switch to the new control plane:
    1. Get the revision label that is on istiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-1139-10-67998f4b55-lrzpz    1/1    Running  0         68m  asm-1129-0
          istiod-asm-1139-10-67998f4b55-r76kr    1/1    Running  0         68m  asm-1129-0
          istiod-1129-0-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1139-10
          istiod-1129-0-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1139-10
    2. Assign the newer revision label to an environment variable.

      In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1139-10

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label to istio-system namespace and remove the istio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  6. Delete the previous versions:
    1. Navigate into the directory where you installed asmcli.
    2. Store the output directory for your Anthos Service Mesh installation in the DIR_PATH environment variable. This is the same directory you specified in the Upgrade with optional features procedure.
      export DIR_PATH=OUTPUT_DIR
    3. Create a shell script containing the following commands:
      #!/bin/bash
      
      set -ex
      
      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      
    4. Execute the script to delete the previous versions.

Outside Google Cloud

These instructions cover upgrading Anthos Service Mesh on:

  • Anthos clusters on VMware (GKE on-prem)
  • Anthos on bare metal
  • Anthos clusters on AWS
  • Amazon EKS

The sequence for upgrading to Anthos Service Mesh version 1.13.9 for your hybrid installation is as follows:

  1. Prepare for the upgrade.
  2. Install the new version of Anthos Service Mesh.
  3. Delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation.
  4. Upgrade your gateways and configure the new webhooks.

Prepare to upgrade Anthos Service Mesh to version 1.13.9

  1. Review the requirements in Upgrade Anthos Service Mesh, but do not perform the upgrade yet.
  2. Before installing the new version, determine the current revision. You will need this information to delete the previous Anthos Service Mesh version's deployments, services, and webhooks from your current installation. Use the following command to store the current istiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
    echo $DELETE_REV

    Your output should look something like 1.12.9-asm.2

  3. Create a new overlay.yaml file or verify that your existing overlay.yaml contains the following contents:
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:  
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: true
          k8s:
            nodeSelector:
              # default node selector, if different or not using node selectors, change accordingly.
              cloud.google.com/gke-nodepool: apigee-runtime
            resources:
              requests:
                cpu: 1000m
            service:
              type: LoadBalancer
              loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
              ports:
                - name: http-status-port
                  port: 15021
                - name: http2
                  port: 80
                  targetPort: 8080
                - name: https
                  port: 443
                  targetPort: 8443
      values:
        gateways:
          istio-ingressgateway:
            runAsRoot: true
    
      meshConfig:
        accessLogFormat:
          '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  4. Follow the instructions in the following sections in the Anthos Service Mesh documentation:
    1. Download asmcli
    2. Grant cluster admin permissions
    3. Validate project and cluster
    4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section".
  5. Switch to the new control plane:
    1. Get the revision label that is on istiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-1139-10-67998f4b55-lrzpz    1/1    Running  0         68m  asm-1129-0
          istiod-asm-1139-10-67998f4b55-r76kr    1/1    Running  0         68m  asm-1129-0
          istiod-1129-0-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1139-10
          istiod-1129-0-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1139-10
    2. Assign the newer revision label to an environment variable.

      In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1139-10

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label to istio-system namespace and remove the istio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  6. Delete the previous versions:
    1. Navigate into the directory where you installed asmcli.
    2. Store the output directory for your Anthos Service Mesh installation in the DIR_PATH environment variable. This is the same directory you specified in the Upgrade with optional features procedure.
      export DIR_PATH=OUTPUT_DIR
    3. Create a shell script containing the following commands:
      #!/bin/bash
      
      set -ex
      
      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl apply -f ${DIR_PATH}/asm/istio/istiod-service.yaml
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      
    4. Execute the script to delete the previous versions.

AKS / EKS

In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version istio-1.13.9-asm.10 on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

  1. Before installing the new version, determine the current revision. You will need this information to delete the validating webhook and mutating webhook from your current Anthos Service Mesh installation. Use the following command to store the current istiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
    echo $DELETE_REV

    Your output should look something like 1.12.9-asm.2

  2. Linux

  3. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-linux-amd64.tar.gz
  4. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-linux-amd64.tar.gz.1.sig
    openssl dgst -verify /dev/stdin -signature istio-1.13.9-asm.10-linux-amd64.tar.gz.1.sig istio-1.13.9-asm.10.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  5. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  6. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  7. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH
  8. Mac OS

  9. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-osx.tar.gz
  10. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-osx.tar.gz.1.sig
    openssl dgst -sha256 -verify /dev/stdin -signature istio-1.13.9-asm.10-osx.tar.gz.1.sig istio-1.13.9-asm.10.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  11. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  12. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  13. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH
  14. Windows

  15. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-win.zip
  16. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.13.9-asm.10-win.zip.1.sig istio-1.13.9-asm.10.win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  17. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-win.zip

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests\profiles directory.
  18. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  19. For convenience, add the tools in the \bin directory to your PATH:
    set PATH=%CD%\bin:%PATH%
  20. Now that Anthos Service Mesh Istio is installed, check the version of istioctl:
    istioctl version
  21. Create a namespace called istio-system for the control plane components:
    kubectl create namespace istio-system

Installing Anthos Service Mesh

  1. Edit your overlay.yaml file or create a new one with the following contents:
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      meshConfig:
        accessLogFile: /dev/stdout
        enableTracing: true
        accessLogFormat:
          '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: true
          k8s:
            service:
              type: LoadBalancer
              ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: http2
                port: 80
                targetPort: 8080
              - name: https
                port: 443
                targetPort: 8443
    
  2. Install Anthos Service Mesh with istioctl using the asm-multicloud profile:
    istioctl install \
        --set profile=asm-multicloud \
        --set revision="asm-1139-10" \
        --filename overlay.yaml

    Your output should look something like:

    kubectl get pods -n istio-system
    NAME                                   READY   STATUS    RESTARTS   AGE
    istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
    istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
    istiod-asm-1139-10-798ffb964-2ls88       1/1     Running   0          3m21s
    istiod-asm-1139-10-798ffb964-fnj8c       1/1     Running   1          3m21s

    The --set revision argument adds a revision label in the format istio.io/rev=asm-1139-10 to istiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particular istiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label on istiod.

  3. Verify that your install completed:
    kubectl get svc -n istio-system

    Your output should look something like:

    NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
    istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
    istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
    istiod-asm-1139-10       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
  4. Switch to the new control plane:
    1. Get the revision label that is on istiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-1139-10-67998f4b55-lrzpz    1/1    Running  0         68m  asm-1129-0
          istiod-asm-1139-10-67998f4b55-r76kr    1/1    Running  0         68m  asm-1129-0
          istiod-1129-0-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1139-10
          istiod-1129-0-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1139-10
    2. Assign the newer revision label to an environment variable.

      In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1139-10

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label to istio-system namespace and remove the istio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  5. Delete the previous versions:
    1. Navigate into the directory where you installed asmcli.
    2. Create a shell script containing the following commands:
      #!/bin/bash
      
      set -ex
      
      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      
    3. Execute the script to delete the previous versions.

OpenShift

In these instructions the process of upgrading Anthos Service Mesh (Anthos Service Mesh) version istio-1.13.9-asm.10 on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

  1. Before installing the new version, determine the current revision. You will need this information to delete the validating webhook and mutating webhook from your current Anthos Service Mesh installation. Use the following command to store the current istiod revision to an environment variable:
    export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
    echo $DELETE_REV

    Your output should look something like 1.12.9-asm.2

  2. Linux

  3. Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  4. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-linux-amd64.tar.gz
  5. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-linux-amd64.tar.gz.1.sig
    openssl dgst -verify /dev/stdin -signature istio-1.13.9-asm.10-linux-amd64.tar.gz.1.sig istio-1.13.9-asm.10.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  6. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  7. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  8. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH
  9. Mac OS

  10. Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  11. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-osx.tar.gz
  12. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-osx.tar.gz.1.sig
    openssl dgst -sha256 -verify /dev/stdin -signature istio-1.13.9-asm.10-osx.tar.gz.1.sig istio-1.13.9-asm.10.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  13. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  14. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  15. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH
  16. Windows

  17. Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
  18. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-win.zip
  19. Download the signature file and use OpenSSL to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.13.9-asm.10-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.13.9-asm.10-win.zip.1.sig istio-1.13.9-asm.10.win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF
  20. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.13.9-asm.10-win.zip

    The command creates an installation directory in your current working directory named istio-1.13.9-asm.10 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests\profiles directory.
  21. Ensure that you're in the Anthos Service Mesh installation's root directory:
    cd istio-1.13.9-asm.10
  22. For convenience, add the tools in the \bin directory to your PATH:
    set PATH=%CD%\bin:%PATH%
  23. Now that Anthos Service Mesh Istio is installed, check the version of istioctl:
    istioctl version
  24. Create a namespace called istio-system for the control plane components:
    kubectl create namespace istio-system

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

  1. Create a file called istiod-service.yaml with the following contents:
    apiVersion: v1
    kind: Service
    metadata:
      name: istiod
      namespace: istio-system
      labels:
        istio.io/rev: asm-1139-10
        app: istiod
        istio: pilot
        release: istio
    spec:
      ports:
        - port: 15010
          name: grpc-xds # plaintext
          protocol: TCP
        - port: 15012
          name: https-dns # mTLS with k8s-signed cert
          protocol: TCP
        - port: 443
          name: https-webhook # validation and injection
          targetPort: 15017
          protocol: TCP
        - port: 15014
          name: http-monitoring # prometheus stats
          protocol: TCP
      selector:
        app: istiod
        istio.io/rev: asm-1139-10
      meshConfig:
        accessLogFormat:
          '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
  2. Use kubectl to apply the validating webhook configuration:
    kubectl apply -f istiod-service.yaml
  3. Verify that the configuration was applied:
    kubectl get svc -n istio-system

    The response should look similar to:

    NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
    istiod   ClusterIP   172.200.18.133   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   22s

Installing Anthos Service Mesh

  1. Edit your overlay.yaml file or create a new one with the following contents:
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      meshConfig:
        accessLogFile: /dev/stdout
        enableTracing: true
        accessLogFormat:
          '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'
      components:
        ingressGateways:
          - name: istio-ingressgateway
            enabled: true
            k8s:
              service:
                type: LoadBalancer
                ports:
                - name: status-port
                  port: 15021
                  targetPort: 15021
                - name: http2
                  port: 80
                  targetPort: 8080
                - name: https
                  port: 443
                  targetPort: 8443
    
  2. Install Anthos Service Mesh with istioctl using the asm-multicloud profile:
    istioctl install \
        --set profile=asm-multicloud \
        --set revision="asm-1139-10" \
        --filename overlayfile.yaml

    Your output should look something like:

    kubectl get pods -n istio-system
    NAME                                   READY   STATUS    RESTARTS   AGE
    istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
    istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
    istiod-asm-1139-10-798ffb964-2ls88       1/1     Running   0          3m21s
    istiod-asm-1139-10-798ffb964-fnj8c       1/1     Running   1          3m21s

    The --set revision argument adds a revision label in the format istio.io/rev=1.6.11-asm.1 to istiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particular istiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label on istiod.

  3. Verify that your install completed:
    kubectl get svc -n istio-system

    Your output should look something like:

    NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
    istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
    istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
    istiod-asm-1139-10       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s
  4. Switch to the new control plane:
    1. Get the revision label that is on istiod:
      kubectl get pod -n istio-system -L istio.io/rev

      The output from the command is similar to the following.

          NAME                                  READY  STATUS  RESTARTS   AGE  REV
          istiod-asm-1139-10-67998f4b55-lrzpz    1/1    Running  0         68m  asm-1129-0
          istiod-asm-1139-10-67998f4b55-r76kr    1/1    Running  0         68m  asm-1129-0
          istiod-1129-0-1-5cd96f88f6-n7tj9      1/1    Running  0         27s  asm-1139-10
          istiod-1129-0-1-5cd96f88f6-wm68b      1/1    Running  0         27s  asm-1139-10
    2. Assign the newer revision label to an environment variable.

      In the output, under the REV column, note the value of the revision label for the new version. In this example, the value is asm-1139-10

      export UPGRADE_REV="REVISION_LABEL"
    3. Add the revision label to istio-system namespace and remove the istio-injection label (if it exists) with the following command.
      kubectl label namespace istio-system istio.io/rev=$UPGRADE_REV istio-injection- --overwrite

      If you see "istio-injection not found" in the output, you can ignore it. That means that the namespace didn't previously have the istio-injection label. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

    4. Restart the Pods to trigger re-injection.
      kubectl rollout restart deployment -n istio-system
    5. Test your application to verify that the workloads are working correctly.
    6. If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
  5. Delete the previous versions:
    1. Navigate into the directory where you installed asmcli.
    2. Create a shell script containing the following commands:
      #!/bin/bash
      
      set -ex
      
      if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
        kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
        kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
      fi
      
    3. Execute the script to delete the previous versions.

Rolling back an upgrade

Follow these steps to roll back a previous upgrade:

  1. Clean up completed jobs for the hybrid runtime namespace, where NAMESPACE is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace is apigee:
    kubectl delete job -n NAMESPACE \
      $(kubectl get job -n NAMESPACE \
      -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  2. Clean up completed jobs for the apigee-system namespace:
    kubectl delete job -n apigee-system \
      $(kubectl get job -n apigee-system \
      -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  3. Change the APIGEECTL_HOME variable to point to the directory that contains the previous version of apigeectl. For example:
    export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
  4. Undo the changes to your overrides file:
    1. Remove or comment out ingressGateways and all its properties.
    2. Set the value of virtualhosts.selector.app to the previous value, for example:
      virtualhosts:
        - name: my-env-group
          selector:
            app: istio-ingressgateway
    3. Remove or comment out ao.args.disableIstioConfigInAPIServer.
  5. In the root directory of the installation you want to roll back to, run apigeectl apply, check the status of your pods, and then run apigeectl init. Be sure to use the original overrides file for the version you wish to roll back to:
    1. In the hybrid-files directory, run apigeectl apply:
      $APIGEECTL_HOME/apigeectl apply -f ORIGINAL_OVERRIDES_FILE

      Where ORIGINAL_OVERRIDES_FILE is relative path and filename of the overrides file for your previous version hybrid installation, for example, ./overrides/overrides1.7.yaml.

    2. Check the status of your pods:
      kubectl -n NAMESPACE get pods

      Where NAMESPACE is your Apigee hybrid namespace.

    3. Check the status of apigeeds:
      kubectl describe apigeeds -n apigee

      Your output should look something like:

      Status:
        Cassandra Data Replication:
        Cassandra Pod Ips:
          10.8.2.204
        Cassandra Ready Replicas:  1
        Components:
          Cassandra:
            Last Successfully Released Version:
              Revision:  v1-f8aa9a82b9f69613
              Version:   v1
            Replicas:
              Available:  1
              Ready:      1
              Total:      1
              Updated:    1
            State:        running
        Scaling:
          In Progress:         false
          Operation:
          Requested Replicas:  0
        State:                 running

      Proceed to the next step only when the apigeeds pod is running.

    4. Run the following command to make note of what your new replica count values will be for message processor after the upgrade.If these values do not match what you have set previously, change the values in your overrides file to match your previous configuration.
      apigeectl apply -f ORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2

      Your output should look something like:

            autoScaler:
              minReplicas: 2
              maxReplicas: 10
    5. If you are rolling back to hybrid v1.8.4 or earlier, delete the controller deployment used by hybrid v1.8.5 and newer:
      kubectl -n apigee-system delete deploy apigee-controller-manager
    6. Run apigeectl init:
      $APIGEECTL_HOME/apigeectl init -f ORIGINAL_OVERRIDES_FILE
  6. Delete the Apigee ingress gateway manager deployment. This component is relevant only to Apigee hybrid versions 1.8 and newer.
    kubectl delete deployment -n NAMESPACE apigee-ingress-gateway-manager

    Where NAMESPACE is your Apigee hybrid namespace.