Preparing for the upgrade
If you customized the previous installation, you need the same customizations
when you upgrade to a new Anthos Service Mesh version or migrate from Istio. If you
customized the installation by adding the --set values
flag to
istioctl install
, you must add those settings to an IstioOperator
YAML file,
referred to as an
overlay file. You specify the overlay
file by using the --custom_overlay
option with the filename when you run the
script. The script passes the overlay file to istioctl install
.
The script follows the revision upgrade
process (referred to as "canary" upgrades in the Istio documentation). With a
revision-based upgrade, the script installs a new revision of the control plane
alongside the existing control plane. When installing the new version,
the script includes a revision
label that identifies the new control plane.
You then migrate to the new version by setting the same revision
label on your
workloads and performing a rolling restart to re-inject the proxies so that they
use the new Anthos Service Mesh version and configuration. With this approach, you can
monitor the effect of the upgrade on a small percentage of your workloads. After
testing your application, you can migrate all traffic to the new version. This
approach is much safer than doing an in-place upgrade where new control plane
components replace the previous version.
Setting up your environment
You need the following tools on the machine you want to install Anthos Service Mesh from. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.
- The
curl
command-line tool. - The Google Cloud CLI.
After installing the Google Cloud CLI:
Authenticate with the Google Cloud CLI:
gcloud auth login
Update the components:
gcloud components update
Install
kubectl
:gcloud components install kubectl
Install the required version of
kpt
:curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v0.39.2/kpt_linux_amd64 > kpt_0_39_2 chmod +x kpt_0_39_2 alias kpt="$(readlink -f kpt_0_39_2)"
Switch context to your user cluster:
kubectl config use-context CLUSTER_NAME
Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:
kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=USER_ACCOUNT
Downloading the installation file
Linux
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-linux-amd64.tar.gz
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-linux-amd64.tar.gz.1.sig openssl dgst -verify /dev/stdin -signature istio-1.10.6-asm.2-linux-amd64.tar.gz.1.sig istio-1.10.6-asm.2-linux-amd64.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.10.6-asm.2-linux-amd64.tar.gz
The command creates an installation directory in your current working directory named
istio-1.10.6-asm.2
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.10.6-asm.2
Mac OS
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-osx.tar.gz
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-osx.tar.gz.1.sig openssl dgst -sha256 -verify /dev/stdin -signature istio-1.10.6-asm.2-osx.tar.gz.1.sig istio-1.10.6-asm.2-osx.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.10.6-asm.2-osx.tar.gz
The command creates an installation directory in your current working directory named
istio-1.10.6-asm.2
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.10.6-asm.2
Windows
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-win.zip
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.10.6-asm.2-win.zip.1.sig openssl dgst -verify - -signature istio-1.10.6-asm.2-win.zip.1.sig istio-1.10.6-asm.2-win.zip <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.10.6-asm.2-win.zip
The command creates an installation directory in your current working directory named
istio-1.10.6-asm.2
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.10.6-asm.2
Preparing resource configuration files
When you run the istioctl install
command, you include the
revisioned-custom-ingressgateway.yaml
file on the command line. This file
allows you to control when you switch to the new version of the
istio-ingressgateway
after upgrading. Do the following steps to
download and configure this file:
Change to the directory where you want to download the
anthos-service-mesh
package.Download the package:
kpt pkg get \ https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git/asm@release-1.10-asm asm
Set the tag to the version of Anthos Service Mesh that you are installing:
kpt cfg set asm anthos.servicemesh.tag 1.10.6-asm.2
Set the validating webhook to use a revision label:
kpt cfg set asm anthos.servicemesh.rev asm-1106-2
When you install Anthos Service Mesh, you set a revision label on
istiod
. You need to set the same revision on the validating webhook.
Upgrading Anthos Service Mesh
To install a new version of Anthos Service Mesh, we recommend that you follow the
revision-based upgrade process (also referred to as a "canary upgrade"). With a
revision-based upgrade, you install a new version of the control plane alongside
the existing control plane. When installing the new version, you include a
revision
label that identifies the version of the new control plane. Each
revision is a full Anthos Service Mesh control plane implementation with its own
Deployment and Service.
You then migrate to the new version by setting the same revision
label on your
workloads to point to the new control plane and performing a rolling restart to
re-inject the proxies with the new Anthos Service Mesh version. With this approach,
you can monitor the effect of the upgrade on a small percentage of your
workloads. After testing your application, you can migrate all traffic to the
new version. This approach is much safer than doing an in-place upgrade where a
new control plane replaces the previous version of the control plane.
Updating the control plane
If needed, change to the
istio-1.10.6-asm.2
directory. Theistioctl
client is version dependent. Make sure that you use the version in theistio-1.10.6-asm.2/bin
directory.Run the following command to deploy the new control plane. If you want to enable a supported optional feature, include
-f
and the YAML filename on the following command line. See Enabling optional features for more information.bin/istioctl install \ --set profile=asm-multicloud \ --set revision=asm-1106-2
The --set revision
argument adds a istio.io/rev
label to istiod
. After
running the command, you have two control plane Deployments and Services
running side-by-side:
kubectl get pods -n istio-system
Deploying and redeploying workloads
Get the revision label that is on
istiod
and theistio-ingressgateway
.kubectl get pod -n istio-system -L istio.io/rev
The output from the command is similar to the following.
NAME READY STATUS RESTARTS AGE REV istio-ingressgateway-65d884685d-6hrdk 1/1 Running 0 67m istio-ingressgateway-65d884685d-94wgz 1/1 Running 0 67m istio-ingressgateway-asm-182-2-8b5fc8767-gk6hb 1/1 Running 0 5s asm-1106-2 istio-ingressgateway-asm-182-2-8b5fc8767-hn4w2 1/1 Running 0 20s asm-1106-2 istiod-asm-176-1-67998f4b55-lrzpz 1/1 Running 0 68m asm-198-3 istiod-asm-176-1-67998f4b55-r76kr 1/1 Running 0 68m asm-198-3 istiod-asm-182-2-5cd96f88f6-n7tj9 1/1 Running 0 27s asm-1106-2 istiod-asm-182-2-5cd96f88f6-wm68b 1/1 Running 0 27s asm-1106-2
In the output, under the
REV
column, note the value of the revision label for the new version. In this example, the value isasm-1106-2
.Also note the value in the revision label for the old
istiod
version. You need this to delete the old version ofistiod
when you finish moving workloads to the new version. In the example output, the value of the revision label for the old version isasm-198-3
.
Switch the
istio-ingressgateway
to the new revision. In the following command, changeREVISION
to the value that matches the revision label of the new version.kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "REVISION"}]'
Expected output:
service/istio-ingressgateway patched
Add the revision label to a namespace and remove the
istio-injection
label (if it exists). In the following command, changeREVISION
to the value that matches the new revision ofistiod
.kubectl label namespace NAMESPACE istio.io/rev=REVISION istio-injection- --overwrite
If you see
"istio-injection not found"
in the output, you can ignore it. That means that the namespace didn't previously have theistio-injection
label. Because auto-injection fails if a namespace has both theistio-injection
and the revision label, allkubectl label
commands in the Anthos Service Mesh documentation include removing theistio-injection
label.Restart the Pods to trigger re-injection.
kubectl rollout restart deployment -n NAMESPACE
Verify that your Pods are configured to point to the new version of
istiod
.kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
Test your application to verify that the workloads are working correctly.
If you have workloads in other namespaces, repeat the steps to label the namespace and restart Pods.
If you are satisfied that your application is working as expected, continue with the steps to transition to the new version of
istiod
. If there's an issue with your application, follow the steps to rollback.Complete the transition
If you are satisfied that your application is working as expected, remove the old control plane to complete the transition to the new version.
Change to the directory where the files from the
anthos-service-mesh
GitHub repository are located.Configure the validating webhook to use the new control plane.
kubectl apply -f asm/istio/istiod-service.yaml
Delete the old
istio-ingressgateway
Deployment. The command that you run depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh:Migrate
If you migrated from Istio, the old
istio-ingressgateway
doesn't have a revision label.kubectl delete deploy/istio-ingressgateway -n istio-system
Upgrade
If you upgraded from a previous Anthos Service Mesh version, in the following command, replace
OLD_REVISION
with the revision label for the previous version of theistio-ingressgateway
.kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=OLD_REVISION -n istio-system --ignore-not-found=true
Delete the old version of
istiod
. The command that you use depends on whether you are migrating from Istio or upgrading from a previous version of Anthos Service Mesh.Migrate
If you migrated from Istio, the old
istio-ingressgateway
doesn't have a revision label.kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true
Upgrade
If you upgraded from a previous Anthos Service Mesh version, in the following command, make sure that
OLD_REVISION
matches the revision label for the previous version ofistiod
.kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-OLD_REVISION -n istio-system --ignore-not-found=true
Remove the old version of the
IstioOperator
configuration.kubectl delete IstioOperator installed-state-OLD_REVISION -n istio-system
The expected output is similar to the following:
istiooperator.install.istio.io "installed-state-OLD_REVISION" deleted
Rollback
If you encountered an issue when testing your application with the new version of
istiod
, follow these steps to rollback to the previous version:Switch back to the old version of the
istio-ingressgateway
. In the following command, replaceOLD_REVISION
with the old revision.kubectl patch service -n istio-system istio-ingressgateway --type='json' -p='[{"op": "replace", "path": "/spec/selector/service.istio.io~1canonical-revision", "value": "OLD_REVISION"}]'
Relabel your namespace to enable auto-injection with the previous version of
istiod
. The command that you use depends on whether you used a revision label oristio-injection=enabled
with the previous version.If you used a revision label for auto-injection:
kubectl label namespace NAMESPACE istio.io/rev=OLD_REVISION --overwrite
If you used
istio-injection=enabled
:kubectl label namespace NAMESPACE istio.io/rev- istio-injection=enabled --overwrite
Expected output:
namespace/NAMESPACE labeled
Confirm that the revision label on the namespace matches the revision label on the previous version of
istiod
:kubectl get ns NAMESPACE --show-labels
Restart the Pods to trigger re-injection so the proxies have the previous version:
kubectl rollout restart deployment -n NAMESPACE
Remove the new
istio-ingressgateway
Deployment. Make sure that the value ofREVISION
in the following command is correct.kubectl delete deploy -l app=istio-ingressgateway,istio.io/rev=REVISION -n istio-system --ignore-not-found=true
Remove the new version of
istiod
. Make sure that the value ofREVISION
in the following command is correct.kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod-REVISION -n istio-system --ignore-not-found=true
Remove the new version of the
IstioOperator
configuration.kubectl delete IstioOperator installed-state-REVISION -n istio-system
Expected output is similar to the following:
istiooperator.install.istio.io "installed-state-REVISION" deleted
If you didn't include the
--disable_canonical_service
flag, the script enabled the Canonical Service controller. We recommend that you leave it enabled, but if you need to disable it, see Enabling and disabling the Canonical Service controller.