Upgrading Apigee hybrid to version 1.7

Upgrading to version 1.7.1 overview.

Upgrading to Apigee hybrid version 1.7 may require downtime:

When upgrading the Apigee controller to version 1.7.1, all Apigee deployments undergo a rolling restart. To minimize downtime in production hybrid environments during a rolling restart, make sure you are running at least two clusters (in the same or different region/data center). Divert all production traffic to a single cluster and take the cluster you are about to upgrade offline, and then proceed with the upgrade process. Repeat the process for each cluster.
If you are upgrading Anthos Service Mesh (ASM) to version 1.12.x, from a version prior to ASM version 1.9, you should expect a few seconds of downtime in the cluster you are upgrading when the ingress changes between the two versions.
If you change your certificate authority while upgrading ASM, you may experience downtime in your ingress gateway. See Migrating to Mesh CA for more information.

The procedures for upgrading Apigee hybrid are organized in the following sections:

Prepare to upgrade to version 1.7.
Install hybrid runtime version 1.7.1.
Upgrade cert-manager.
Upgrade ASM.

Prerequisite

Apigee hybrid version 1.6. If you are updating from an earlier version see the instructions for Upgrading Apigee hybrid to version 1.6.

Prepare to upgrade to version 1.7

Backup your hybrid installation

These instructions use the environment variable APIGEECTL_HOME for the directory in your file system where you have installed apigeectl. If needed, cd into your apigeectl directory and define the variable with the following command:
Linux
```
export APIGEECTL_HOME=$PWD
```
```
echo $APIGEECTL_HOME
```
Mac OS
```
export APIGEECTL_HOME=$PWD
```
```
echo $APIGEECTL_HOME
```
Windows
```
set APIGEECTL_HOME=%CD%
```
```
echo %APIGEECTL_HOME%
```
(Recommended) Make a backup copy of your version 1.6 $APIGEECTL_HOME/ directory. For example:
```
tar -czvf $APIGEECTL_HOME/../apigeectl-v1.6-backup.tar.gz $APIGEECTL_HOME
```
(Recommended) Backup your Cassandra database following the instructions in Cassandra backup and recovery

Upgrade your Kubernetes version

Upgrade your Kubernetes platform to the versions supported by hybrid 1.7. Follow your platform's documentation if you need help.

Click to expand a list of supported platforms

	Apigee hybrid versions
Platforms	1.5	1.6	1.7
Anthos⁽³⁾ (Google Cloud)	1.18.x 1.19.x	1.19.x 1.20.x 1.21.x	1.20.x 1.21.x 1.22.x⁽⁶⁾ 1.23.x⁽⁶⁾
Anthos⁽³⁾ (AWS)	1.6.x 1.7.x 1.8.x⁽¹⁾	1.7.x 1.8.x⁽¹⁾ 1.9.3+ 1.10.x	1.9.x 1.10.x
Anthos⁽³⁾ (Azure)	N/A	1.8.x	1.8.x
Anthos⁽³⁾ (on-premises - VMware)	1.6.x 1.7.x 1.8.x⁽¹⁾	1.7.x 1.8.x⁽¹⁾ 1.9.3+ 1.10.x	1.9.x 1.10.x
Anthos⁽³⁾ (Bare Metal)	1.6.x 1.7.x	1.7.x 1.8.2+ 1.9.3+ 1.10.x	1.9.x 1.10.x 1.11.x⁽⁶⁾
Anthos⁽³⁾ (attached clusters - multi-cloud) Kubernetes version (EKS)	1.18.x 1.19.x	1.19.x 1.20.x 1.21.x	1.21.x 1.22.x⁽⁶⁾ 1.23.x⁽⁶⁾
Anthos⁽³⁾ (attached clusters - multi-cloud) Kubernetes version (AKS)	1.18.x 1.19.x	1.19.x 1.20.x 1.21.x	1.21.x 1.22.x⁽⁶⁾ 1.23.x⁽⁶⁾
Anthos⁽³⁾ (attached clusters - multi-cloud) OpenShift version	4.5 4.6 4.7	4.6 4.7 4.8	4.7 4.8
Anthos⁽³⁾ (attached clusters - multi-cloud) Konvoy version	1.7.x	1.7.x	N/A
Components	1.5	1.6	1.7
Anthos Service Mesh (ASM)⁽²⁾	1.7.x 1.8.x 1.9.x 1.12.x⁽⁴⁾	1.9.x 1.10.x 1.12.x⁽⁵⁾	1.10.x 1.11.x 1.12.x 1.13.x⁽⁶⁾
JDK	JDK 11	JDK 11	JDK 11
cert-manager	1.2.0	1.5.4	1.7.x
Cassandra	3.11.6	3.11.6	3.11.9
⁽¹⁾ On version 1.8.2 and above, follow the instructions in this document: Conflict with cert-manager when upgrading to version 1.8.2 or above. ⁽²⁾ Install only Supported versions of ASM. ⁽³⁾ Install only Supported versions of Anthos. ⁽⁴⁾ Support available with Apigee hybrid version 1.5, starting with version 1.5.9. ⁽⁵⁾ Support available with Apigee hybrid version 1.6.6 and newer. ⁽⁶⁾ Support available with Apigee hybrid version 1.7.2 and newer.

Add the Cloud Trace Agent role to the `apigee-runtime` service account

Optional: If you plan to use Cloud trace, ensure your apigee-runtime service account has the Cloud Trace Agent (roles/cloudtrace.agent) Google role. You can do so in the Cloud console > IAM & Admin > Service accounts UI or with the following commands:

Get the email address for your apigee-runtime service account with the following command:
```
gcloud iam service-accounts list --filter "apigee-runtime"
```
If it matches the pattern apigee-runtime@$ORG_NAME.iam.gserviceaccount.com, you can use that pattern in the next step.

Assign the Cloud Trace Agent role to the service account:

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/cloudtrace.agent"

Where: $PROJECT_ID is the name of your Google Cloud project

Replace `metrics:stackdriverExporter` properties in your overrides.

Note: This step is only necessary if you have specified metrics:stackdriverExporter in your overrides.yaml file.

Starting in Hybrid version 1.7, metrics:stackdriverExporter has been replaced with metrics:appStackdriverExporter and metrics:proxyStackdriverExporter. Replace those properties with equivalent properties. For example, replace:

metrics:
  ... ...
  stackdriverExporter:
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 128m
        memory: 512Mi

with:

metrics:
  ... ...
  appStackdriverExporter:
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 128m
        memory: 512Mi
  proxyStackdriverExporter:
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 128m
        memory: 512Mi

See the Configuration property reference: metrics

Install the hybrid 1.7.1 runtime

Store the latest version number in a variable using the following command:

Linux

export VERSION=$(curl -s \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt?ignoreCache=1)

Mac OS

export VERSION=$(curl -s \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt)

Windows

for /f "tokens=*" %a in ('curl -s ^
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/current-version.txt') ^
do set VERSION=%a

Check that the variable was populated with a version number using the following command. If you want to use a different version, you can save that in an environment variable instead.
```
echo $VERSION
```
```
  1.7.1
```

Download the release package for your operating system using the following command:

Linux

Linux 64 bit:

curl -LO \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/$VERSION/apigeectl_linux_64.tar.gz

Mac OS

Mac 64 bit:

curl -LO \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/$VERSION/apigeectl_mac_64.tar.gz

Windows

Windows 64 bit:

curl -LO ^
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/%VERSION%/apigeectl_windows_64.zip

Rename your current apigeectl/ directory to a backup directory name. For example:

Linux

mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.6/

Mac OS

mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.6/

Windows

rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.6

Extract the downloaded gzip file contents into your hybrid base directory using the following command:

Linux

tar xvzf filename.tar.gz -C path-to-base-directory

Mac OS

tar xvzf filename.tar.gz -C path-to-base-directory

Windows

tar xvzf filename.zip -C path-to-base-directory

Change directory to the base directory.
The tar contents are, by default, expanded into a directory with the version and platform in its name. For example: ./apigeectl_1.7.1-d591b23_linux_64. Rename that directory to apigeectl using the following command:
Linux
```
mv apigeectl_1.7.1-d591b23_linux_64 apigeectl
```
Mac OS
```
mv apigeectl_1.7.1-d591b23_mac_64 apigeectl
```
Windows
```
rename apigeectl_1.7.1-d591b23_windows_64 apigeectl
```
Change to the directory using the following command:
```
cd ./apigeectl
```
This directory is the apigeectl home directory. It is where the apigeectl executable command is located.
Verify the version of apigeectl with the version command:
```
./apigeectl version
```
```
Version: 1.7.1
```
Create an environment variable to hold this home directory path using the following command:
```
export APIGEECTL_HOME=$PWD
```
Verify that the variable holds the correct path using the following command:
```
echo $APIGEECTL_HOME
```
In the new apigeectl/ directory, run apigeectl init, apigeectl apply, and apigeectl check-ready:
1. Initialize hybrid 1.7.1:
```
apigeectl init -f OVERRIDES.yaml
```
  Where OVERRIDES.yaml is your edited overrides.yaml file.
2. Check that it initialized correctly with the following commands:
```
apigeectl check-ready -f OVERRIDES.yaml
```
```
kubectl describe apigeeds -n apigee
```
  Your output should look something like:
```
Status:
  Cassandra Data Replication:
  Cassandra Pod Ips:
    10.8.2.204
  Cassandra Ready Replicas:  1
  Components:
    Cassandra:
      Last Successfully Released Version:
        Revision:  v1-f8aa9a82b9f69613
        Version:   v1
      Replicas:
        Available:  1
        Ready:      1
        Total:      1
        Updated:    1
      State:        running
  Scaling:
    In Progress:         false
    Operation:
    Requested Replicas:  0
  State:                 running
```
3. Check for errors with a dry run:
```
apigeectl apply -f OVERRIDES.yaml --dry-run=client
```
4. Apply your overrides. Select and follow the instructions for production environments or demo/experimental environments, depending on your installation.
  During the upgrade process each component will perform a rolling restart. Therefore, for production environments, it is best to apply the upgrade to one component at a time.
  Production
  
  For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.
  1. Apply your overrides to upgrade Cassandra:
    apigeectl apply -f OVERRIDES.yaml --datastore
  2. Check completion:
    apigeectl check-ready -f OVERRIDES.yaml
    
    Tip: If check-ready fails, you can get more information about your pods with:
    kubectl -n NAMESPACE get pods
    
    Where NAMESPACE is your Apigee hybrid namespace.
    
    Proceed to the next step only when the pods are ready.
  3. Apply your overrides to upgrade Telemetry components and check completion:
    apigeectl apply -f OVERRIDES.yaml --telemetry
    
    apigeectl check-ready -f OVERRIDES.yaml
  4. Bring up Redis components:
    apigeectl apply -f OVERRIDES.yaml --redis
  5. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
    apigeectl apply -f OVERRIDES.yaml --org
    
    apigeectl check-ready -f OVERRIDES.yaml
  6. Apply your overrides to upgrade your environments. You have two choices:
    
    Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
    apigeectl apply -f OVERRIDES.yaml --env ENV_NAME
    
    apigeectl check-ready -f OVERRIDES.yaml
    
    Where ENV_NAME is the name of the environment you are upgrading.
    
    All environments at one time: Apply your overrides to all environments at once and check completion:
    apigeectl apply -f OVERRIDES.yaml --all-envs
    
    apigeectl check-ready -f OVERRIDES.yaml
  Demo/Experimental
  
  In most demo or experimental environments, you can apply the overrides to all components at once. If your demo/experimental environment large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments
  1. apigeectl apply -f OVERRIDES.yaml
  2. Check the status:
    apigeectl check-ready -f OVERRIDES.yaml

Upgrade cert-manager to version v1.7.2

If you are running a version of cert-manager prior to v1.7.2, you need to upgrade it to v1.7.2.

Check the current cert-manager version using the following command:

kubectl -n cert-manager get deployment -o yaml | grep 'image:'

Something similar to the following is returned:

image: quay.io/jetstack/cert-manager-controller:v1.7.2
image: quay.io/jetstack/cert-manager-cainjector:v1.7.2
image: quay.io/jetstack/cert-manager-webhook:v1.7.2

Remove the deployments using the following command:

$ kubectl delete -n cert-manager deployment cert-manager cert-manager-cainjector cert-manager-webhook

Upgrade cert-manager to v1.7.2 version using the following command:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml

Upgrade ASM to version 1.12

Perform the upgrade using the ASM documentation appropriate for your platform:

The instructions to install and configure ASM are different depending on your platform. The platforms are divided into the following categories:

GKE: Google Kubernetes Engine clusters running on Google Cloud.
Outside Google Cloud: Anthos clusters running on:
- Anthos clusters on VMware (GKE on-prem)
- Anthos on bare metal
- Anthos clusters on AWS
- Amazon EKS
Other Kubernetes Platforms: Conformant clusters created and running on:
- AKS
- EKS
- OpenShift

GKE

The sequence for upgrading to ASM version 1.12.6 for your hybrid installation is as follows:

Prepare for the upgrade.
Install the new version of ASM.
Delete the previous ASM version's deployments, services, and webhooks from your current installation.
Upgrade your gateways and configure the new webhooks.

To upgrade to ASM version 1.12.6 for hybrid on GKE:

Review the requirements in Upgrade Anthos Service Mesh, but do not perform the upgrade yet.
Before installing the new version, determine the current revision. You will need this information to delete the previous ASM version's deployments, services, and webhooks from your current installation. Use the following command to store the current istiod revision to an environment variable:
```
export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[].metadata.labels.'istio\.io\/rev'}'{"\n"}')
echo ${DELETE_REV}
```

Create a new overlay.yaml file or verify that your existing overlay.yaml contains the following contents:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  revision: asm-1127-2
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          nodeSelector:
            # default node selector, if different or not using node selectors, change accordingly.
            cloud.google.com/gke-nodepool: apigee-runtime
          resources:
            requests:
              cpu: 1000m
          service:
            type: LoadBalancer
            loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
            ports:
              - name: http-status-port
                port: 15021
              - name: http2
                port: 80
                targetPort: 8080
              - name: https
                port: 443
                targetPort: 8443
  meshConfig:
    accessLogFormat:
      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'

Follow the instructions in the following sections in the ASM documentation:
Important: Make sure to follow the instructions to upgrade ASM with optional features, and to include your overlay.yaml.
1. Download asmcli
2. Grant cluster admin permissions
3. Validate project and cluster
4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section"

Delete the mutating webhook and validating webhook:

cd into the directory where you installed asmcli.
Store the current new revision in an environment variable to use in the script to delete the webhooks:
```
UPGRADE_REV="asm-1127-2"
```

create a shell script containing the following commands:

#!/bin/bash

set -ex

PROJECT_ID="YOUR_PROJECT_ID"
CLUSTER_NAME="YOUR_CLUSTER_NAME"
CLUSTER_LOCATION="YOUR_CLUSTER_LOCATION"

kubectl label namespace istio-system istio.io/rev=${UPGRADE_REV} istio-injection- --overwrite
kubectl rollout restart deployment -n istio-system
kubectl apply -n istio-system -f PATH_TO_INGRESSGATEWAYistio-ingressgateway
kubectl apply -n istio-system -f PATH_TO_INGRESSGATEWAY/istio-ingressgateway-connectors

if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
  kubectl apply -f out/asm/istio/istiod-service.yaml
  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
fi

Execute the script to delete the current webhooks.

Follow the steps in Upgrade gateways to create the new webhooks and switch traffic to the new gateways.

Outside Google Cloud

These instructions cover upgrading ASM on:

Anthos clusters on VMware (GKE on-prem)
Anthos on bare metal
Anthos clusters on AWS
Amazon EKS

The sequence for upgrading to ASM version 1.12.6 for your hybrid installation is as follows:

Prepare for the upgrade.
Install the new version of ASM.
Delete the previous ASM version's deployments, services, and webhooks from your current installation.
Upgrade your gateways and configure the new webhooks.

Review the requirements in Upgrade Anthos Service Mesh, but do not perform the upgrade yet.
Before installing the new version, determine the current revision. You will need this information to delete the validating webhook and mutating webhook from your current ASM installation. Use the following command to store the current istiod revision to an environment variable:
```
export DELETE_REV=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[].metadata.labels.'istio\.io\/rev'}'{"\n"}')
echo ${DELETE_REV}
```

Create a new overlay.yaml file or verify that your existing overlay.yaml contains the following contents:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  revision: asm-1127-2
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          nodeSelector:
            # default node selector, if different or not using node selectors, change accordingly.
            cloud.google.com/gke-nodepool: apigee-runtime
          resources:
            requests:
              cpu: 1000m
          service:
            type: LoadBalancer
            loadBalancerIP: STATIC_IP # If you do not have a reserved static IP, leave this out.
            ports:
              - name: http-status-port
                port: 15021
              - name: http2
                port: 80
                targetPort: 8080
              - name: https
                port: 443
                targetPort: 8443
  values:
    gateways:
      istio-ingressgateway:
        runAsRoot: true

  meshConfig:
    accessLogFormat:
      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'

Follow the instructions in the following sections in the ASM documentation:
Important: Make sure to follow the instructions to upgrade ASM with optional features, and to include your overlay.yaml.
1. Download asmcli
2. Grant cluster admin permissions
3. Validate project and cluster
4. Upgrade with optional features. Stop before starting the "Upgrade Gateways section"

Delete the mutating webhook and validating webhook:

cd into the directory where you installed asmcli.
Store the current new revision in an environment variable to use in the script to delete the webhooks:
```
UPGRADE_REV="asm-1127-2"
```

create a shell script containing the following commands:

#!/bin/bash

set -ex

PROJECT_ID="YOUR_PROJECT_ID"
CLUSTER_NAME="YOUR_CLUSTER_NAME"
CLUSTER_LOCATION="YOUR_CLUSTER_LOCATION"


gcloud config configurations activate ${PROJECT_ID}
gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${CLUSTER_LOCATION} --project ${PROJECT_ID}


kubectl label namespace istio-system istio.io/rev=${UPGRADE_REV} istio-injection- --overwrite
kubectl rollout restart deployment -n istio-system
kubectl apply -n istio-system -f PATH_TO_INGRESSGATEWAYistio-ingressgateway
kubectl apply -n istio-system -f PATH_TO_INGRESSGATEWAY/istio-ingressgateway-connectors

if [[ "${DELETE_REV}" != "${UPGRADE_REV}" ]]; then
  kubectl apply -f out/asm/istio/istiod-service.yaml
  kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete deploy -l app=istio-ingressgateway-connectors,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete ValidatingWebhookConfiguration -l app=istiod,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete MutatingWebhookConfiguration -l app=sidecar-injector,istio.io/rev=${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-${DELETE_REV} -n istio-system --ignore-not-found=true
  kubectl delete IstioOperator installed-state-${DELETE_REV} -n istio-system --ignore-not-found=true
fi

Execute the script to delete the current webhooks.

Follow the steps in Upgrade gateways to create the new webhooks and switch traffic to the new gateways.

AKS / EKS

In these instructions the process of upgrading Anthos Service Mesh (ASM) version istio-1.12.7-asm.2 on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-linux-amd64.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-linux-amd64.tar.gz.1.sig

openssl dgst -verify /dev/stdin -signature istio-1.12.7-asm.2-linux-amd64.tar.gz.1.sig istio-1.12.7-asm.2.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-linux-amd64.tar.gz

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the /bin directory to your PATH:

export PATH=$PWD/bin:$PATH

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-osx.tar.gz.1.sig

openssl dgst -sha256 -verify /dev/stdin -signature istio-1.12.7-asm.2-osx.tar.gz.1.sig istio-1.12.7-asm.2.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the /bin directory to your PATH:

export PATH=$PWD/bin:$PATH

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-win.zip.1.sig

openssl dgst -verify - -signature istio-1.12.7-asm.2-win.zip.1.sig istio-1.12.7-asm.2.win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-win.zip

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests\profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the \bin directory to your PATH:

set PATH=%CD%\bin:%PATH%

Now that ASM Istio is installed, check the version of istioctl:
```
istioctl version
```
Create a namespace called istio-system for the control plane components:
```
kubectl create namespace istio-system
```

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

Create a file called istiod-service.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: istiod
  namespace: istio-system
  labels:
    istio.io/rev: asm-1127-2
    app: istiod
    istio: pilot
    release: istio
spec:
  ports:
    - port: 15010
      name: grpc-xds # plaintext
      protocol: TCP
    - port: 15012
      name: https-dns # mTLS with k8s-signed cert
      protocol: TCP
    - port: 443
      name: https-webhook # validation and injection
      targetPort: 15017
      protocol: TCP
    - port: 15014
      name: http-monitoring # prometheus stats
      protocol: TCP
  selector:
    app: istiod
    istio.io/rev: asm-1127-2
  meshConfig:
    accessLogFormat:
      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'

Use kubectl to apply the validating webhook configuration:
```
kubectl apply -f istiod-service.yaml
```

Verify that the configuration was applied:

kubectl get svc -n istio-system

The response should look similar to:

NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
istiod   ClusterIP   172.200.18.133   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   22s

Installing Anthos Service Mesh

Install Anthos Service Mesh with istioctl using the asm-multicloud profile:

istioctl install \
    --set profile=asm-multicloud \
    --set revision="asm-1127-2"

Your output should look something like:

kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
istiod-asm-1127-2-798ffb964-2ls88       1/1     Running   0          3m21s
istiod-asm-1127-2-798ffb964-fnj8c       1/1     Running   1          3m21s

The --set revision argument adds a revision label in the format istio.io/rev=asm-1127-2 to istiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particular istiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label on istiod.

Verify that your install completed:

kubectl get svc -n istio-system

Your output should look something like:

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
istiod-asm-1127-2       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s

OpenShift

In these instructions the process of upgrading Anthos Service Mesh (ASM) version istio-1.12.7-asm.2 on Anthos attached clusters is the same as performing a fresh install.

Preparing to install Anthos Service Mesh

Linux

Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:

oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-linux-amd64.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-linux-amd64.tar.gz.1.sig

openssl dgst -verify /dev/stdin -signature istio-1.12.7-asm.2-linux-amd64.tar.gz.1.sig istio-1.12.7-asm.2.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-linux-amd64.tar.gz

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the /bin directory to your PATH:

export PATH=$PWD/bin:$PATH

Mac OS

Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:

oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-osx.tar.gz.1.sig

openssl dgst -sha256 -verify /dev/stdin -signature istio-1.12.7-asm.2-osx.tar.gz.1.sig istio-1.12.7-asm.2.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the /bin directory to your PATH:

export PATH=$PWD/bin:$PATH

Windows

Grant the anyuid security context constraint (SCC) to the istio-system with the following OpenShift CLI (oc) command:

oc adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.12.7-asm.2-win.zip.1.sig

openssl dgst -verify - -signature istio-1.12.7-asm.2-win.zip.1.sig istio-1.12.7-asm.2.win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.12.7-asm.2-win.zip

The command creates an installation directory in your current working directory named istio-1.12.7-asm.2 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the manifests\profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory:

cd istio-1.12.7-asm.2

For convenience, add the tools in the \bin directory to your PATH:

set PATH=%CD%\bin:%PATH%

Now that ASM Istio is installed, check the version of istioctl:
```
istioctl version
```
Create a namespace called istio-system for the control plane components:
```
kubectl create namespace istio-system
```

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

Create a file called istiod-service.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: istiod
  namespace: istio-system
  labels:
    istio.io/rev: asm-1127-2
    app: istiod
    istio: pilot
    release: istio
spec:
  ports:
    - port: 15010
      name: grpc-xds # plaintext
      protocol: TCP
    - port: 15012
      name: https-dns # mTLS with k8s-signed cert
      protocol: TCP
    - port: 443
      name: https-webhook # validation and injection
      targetPort: 15017
      protocol: TCP
    - port: 15014
      name: http-monitoring # prometheus stats
      protocol: TCP
  selector:
    app: istiod
    istio.io/rev: asm-1127-2
  meshConfig:
    accessLogFormat:
      '{"start_time":"%START_TIME%","remote_address":"%DOWNSTREAM_DIRECT_REMOTE_ADDRESS%","user_agent":"%REQ(USER-AGENT)%","host":"%REQ(:AUTHORITY)%","request":"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%","request_time":"%DURATION%","status":"%RESPONSE_CODE%","status_details":"%RESPONSE_CODE_DETAILS%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","upstream_address":"%UPSTREAM_HOST%","upstream_response_flags":"%RESPONSE_FLAGS%","upstream_response_time":"%RESPONSE_DURATION%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_cluster":"%UPSTREAM_CLUSTER%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","request_method":"%REQ(:METHOD)%","request_path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","request_protocol":"%PROTOCOL%","tls_protocol":"%DOWNSTREAM_TLS_VERSION%","request_id":"%REQ(X-REQUEST-ID)%","sni_host":"%REQUESTED_SERVER_NAME%","apigee_dynamic_data":"%DYNAMIC_METADATA(envoy.lua)%"}'

Use kubectl to apply the validating webhook configuration:
```
kubectl apply -f istiod-service.yaml
```

Verify that the configuration was applied:

kubectl get svc -n istio-system

The response should look similar to:

NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
istiod   ClusterIP   172.200.18.133   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   22s

Installing Anthos Service Mesh

Install Anthos Service Mesh with istioctl using the asm-multicloud profile:

istioctl install \
    --set profile=asm-multicloud \
    --set revision=istio-1.12.7-asm.2

Your output should look something like:

kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-88b6fd976-flgp2   1/1     Running   0          3m13s
istio-ingressgateway-88b6fd976-p5dl9   1/1     Running   0          2m57s
istiod-asm-1127-2-798ffb964-2ls88       1/1     Running   0          3m21s
istiod-asm-1127-2-798ffb964-fnj8c       1/1     Running   1          3m21s

The --set revision argument adds a revision label in the format istio.io/rev=1.6.11-asm.1 to istiod. The revision label is used by the automatic sidecar injector webhook to associate injected sidecars with a particular istiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label on istiod.

Verify that your install completed:

kubectl get svc -n istio-system

Your output should look something like:

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   172.200.48.52    34.74.177.168   15021:30479/TCP,80:30030/TCP,443:32200/TCP,15012:32297/TCP,15443:30244/TCP   3m35s
istiod                 ClusterIP      172.200.18.133   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4m46s
istiod-asm-1127-2       ClusterIP      172.200.63.220   <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3m43s

Rolling back an upgrade

Follow these steps to roll back a previous upgrade:

Clean up completed jobs for the hybrid runtime namespace, where NAMESPACE is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace is apigee:
```
kubectl delete job -n NAMESPACE \
  $(kubectl get job -n NAMESPACE \
  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
```

Clean up completed jobs for the apigee-system namespace:

kubectl delete job -n apigee-system \
  $(kubectl get job -n apigee-system \
  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')

Change the APIGEECTL_HOME variable to point to the directory that contains the previous version of apigeectl. For example:
```
export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
```

In the root directory of the installation you want to roll back to, run apigeectl apply, check the status of your pods, and then run apigeectl init. Be sure to use the original overrides file for the version you wish to roll back to:

Run apigeectl apply:

$APIGEECTL_HOME/apigeectl apply -f overrides/ORIGINAL_OVERRIDES.yaml

Check the status of your pods:
```
kubectl -n NAMESPACE get pods
```
Where NAMESPACE is your Apigee hybrid namespace.

Check the status of apigeeds:

kubectl describe apigeeds -n apigee

Your output should look something like:

Status:
  Cassandra Data Replication:
  Cassandra Pod Ips:
    10.8.2.204
  Cassandra Ready Replicas:  1
  Components:
    Cassandra:
      Last Successfully Released Version:
        Revision:  v1-f8aa9a82b9f69613
        Version:   v1
      Replicas:
        Available:  1
        Ready:      1
        Total:      1
        Updated:    1
      State:        running
  Scaling:
    In Progress:         false
    Operation:
    Requested Replicas:  0
  State:                 running

Proceed to the next step only when the apigeeds pod is running.

Run apigeectl init:

$APIGEECTL_HOME/apigeectl init -f overrides/ORIGINAL_OVERRIDES.yaml

Upgrading Apigee hybrid to version 1.7

Upgrading to version 1.7.1 overview.

Prerequisite

Prepare to upgrade to version 1.7

Backup your hybrid installation

Linux

Mac OS

Windows

Upgrade your Kubernetes version

Click to expand a list of supported platforms

Platforms

Components

Add the Cloud Trace Agent role to the apigee-runtime service account

Replace metrics:stackdriverExporter properties in your overrides.

Install the hybrid 1.7.1 runtime

Linux

Mac OS

Windows

Linux

Mac OS

Windows

Linux

Mac OS

Windows

Linux

Mac OS

Windows

Linux

Mac OS

Windows

Production

Demo/Experimental

Upgrade cert-manager to version v1.7.2

Upgrade ASM to version 1.12

GKE

Outside Google Cloud

AKS / EKS

Preparing to install Anthos Service Mesh

Linux

Mac OS

Windows

Configure the validating webhook

Installing Anthos Service Mesh

OpenShift

Preparing to install Anthos Service Mesh

Linux

Mac OS

Windows

Configure the validating webhook

Installing Anthos Service Mesh

Rolling back an upgrade

Add the Cloud Trace Agent role to the `apigee-runtime` service account

Replace `metrics:stackdriverExporter` properties in your overrides.