Installing Anthos Service Mesh on premises

This guide explains how to do a clean installation of Anthos Service Mesh version 1.8.6-asm.8 on GKE on VMware. If you have a previous version of Anthos Service Mesh installed, refer to Upgrading Anthos Service Mesh GKE on VMware. The installation enables the supported features on your cluster. This guide refers to the cluster as cluster1, but you can repeat these steps to set up many clusters.

Note: The Anthos Service Mesh managed components, Anthos Service Mesh certificate authority and Mesh telemetry, aren't currently supported on GKE on VMware.

About the control plane components

GKE on VMware comes with the following Istio components preinstalled:

• The Anthos Service Mesh certificate authority (Citadel) is installed in the kube-system namespace.
• Pilot and the Istio Ingress Gateway are installed in the gke-system namespace.

GKE on VMware uses these components to enable ingress and to secure communication between Google-controlled components. If you only need ingress functionality, you don't need to install OSS Istio or Anthos Service Mesh. For more information on configuring ingress, see Enabling ingress.

When you install Anthos Service Mesh, its components are installed in the istio-system namespace. Because the Anthos Service Mesh components are in a different namespace, they don't conflict with the GKE on VMware preinstalled Istio components.

Before you begin

Review the following requirements before you begin setup.

Requirements

• You must have an Anthos subscription. Alternatively, a pay-as-you-go billing option is available for GKE Enterprise on Google Cloud only. For more information, see the GKE Enterprise Pricing guide.

• Verify that your user cluster that you install Anthos Service Mesh on has at least 4 vCPUs, 15 GB memory, and 4 nodes.

• You must name your service ports using the following syntax: name: protocol[-suffix] where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.

• Verify that your cluster version is listed in Supported environments. To check your cluster version, you can use the gkectl command line tool. If you don't have gkectl installed, see GKE on-prem downloads.

  gkectl version

Setting up your environment

You need the following tools on the computer where you control the installation process. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.

• The curl command-line tool.
• The Google Cloud CLI.

After installing the Google Cloud CLI:

1. Authenticate with the Google Cloud CLI:

   gcloud auth login
   

2. Update the components:

   gcloud components update
   

3. Install kubectl:

   gcloud components install kubectl
   

4. If you want to deploy and test your installation with the Online Boutique sample application, install kpt:

   gcloud components install kpt
   

Setting environment variables

1. Get the context name for the cluster by using the values under the NAME column in the output of this command:

   kubectl config get-contexts

2. Set the environment variables to the cluster context name, which this guide uses in many steps later:

   export CTX_CLUSTER1=CLUSTER1_CONTEXT_NAME

Granting cluster admin permissions

1. Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

   kubectl --context="${CTX_CLUSTER1}" create clusterrolebinding cluster-admin-binding \
     --clusterrole=cluster-admin \
     --user=USER_ACCOUNT

Downloading the installation file

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-linux-amd64.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-linux-amd64.tar.gz.1.sig
openssl dgst -verify /dev/stdin -signature istio-1.8.6-asm.8-linux-amd64.tar.gz.1.sig istio-1.8.6-asm.8-linux-amd64.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-linux-amd64.tar.gz

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory.

cd istio-1.8.6-asm.8

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.8.6-asm.8-osx.tar.gz.1.sig istio-1.8.6-asm.8-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory.

cd istio-1.8.6-asm.8

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-win.zip.1.sig
openssl dgst -verify - -signature istio-1.8.6-asm.8-win.zip.1.sig istio-1.8.6-asm.8-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-win.zip

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

• Sample applications in the samples directory.
• The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
• The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory.

cd istio-1.8.6-asm.8

Configuring the certificate authority

This section explains how to generate certificates and keys that Anthos Service Mesh on-prem uses to sign your workloads.

For the best security, we highly recommend maintaining an offline root CA and using the subordinate CAs to issue CAs for each cluster. For more information, see Plug in CA Certificates. In this configuration, all workloads in the service mesh use the same root certificate authority (CA). Each Anthos Service Mesh CA uses an intermediate CA signing key and certificate, signed by the root CA. When multiple CAs exist within a mesh, this establishes a hierarchy of trust among the CAs. You can repeat these steps to provision certificates and keys for any number of certificate authorities.

Important: These steps create certificates and keys using a make command that we provide for illustrative purposes only. For the best security, use an offline secure computer and production-ready tools, for example using HashiCorp Vault to securely generate and protect the CA signing key.

1. Create a directory for the certificates and keys:

   mkdir -p certs && \
   pushd certs

2. Generate a root certificate and key:

   make -f ../tools/certs/Makefile.selfsigned.mk root-ca
   

   This generates these files:

   • root-cert.pem: the root certificate
   • root-key.pem: the root key
   • root-ca.conf: the configuration for openssl to generate the root certificate
   • root-cert.csr: the CSR for the root certificate

3. Generate an intermediate certificate and key:

   make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts

   This generates these files in a directory named cluster1:

   • ca-cert.pem: the intermediate certificates
   • ca-key.pem: the intermediate key
   • cert-chain.pem: the certificate chain which istiod uses
   • root-cert.pem: the root certificate
   Note: You can replace cluster1 with a different name. For example, make mycluster-cacerts creates a directory named mycluster.

   If you perform these steps using an offline computer, copy the generated directory to a computer with access to the clusters.

4. Create a secret cacerts including all the input files ca-cert.pem, ca- key.pem, root-cert.pem and cert-chain.pem:

   kubectl --context="${CTX_CLUSTER1}" create namespace istio-system
   kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-system \
     --from-file=cluster1/ca-cert.pem \
     --from-file=cluster1/ca-key.pem \
     --from-file=cluster1/root-cert.pem \
     --from-file=cluster1/cert-chain.pem

   Anthos Service Mesh on-prem will detect the presence of these certificates/keys and use them during the install process in later steps.

5. Return to the previous directory:

   popd

Installing Anthos Service Mesh

1. Create an environment variable for the project ID:

   export PROJECT_ID=YOUR_PROJECT_ID

2. Create an environment variable for the project number:

   export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")

3. Create an environment variable for the mesh identifier. This can be any string, but should be in a format that is consistent among your clusters.

   export MESH_ID="proj-${PROJECT_NUMBER}"

4. Create the configuration for the cluster's control plane, which will install Anthos Service Mesh using the asm-multicloud profile. If you want to enable a supported optional feature, include -f and the YAML filename on the following command line. See Enabling optional features for more information.

   In the following example, use the MESH_ID defined in the previous steps.

   cat <<EOF > cluster.yaml
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     profile: asm-multicloud
     revision: asm-186-8
     values:
       global:
         meshID: MESH_ID
         multiCluster:
           clusterName: CLUSTER_NAME
         network: NETWORK_ID
   EOF
   

   Note: The revision value adds a revision label in the format asm-186-8 to istiod. The automatic sidecar injector webhook uses the revision label to associate injected sidecars with a particular istiod revision. To enable sidecar auto-injection for a namespace, you must label it with a revision that matches the label on istiod.

5. If needed, change to the istio-1.8.6-asm.8 directory. The istioctl client is version dependent. Make sure that you use the version in the istio-1.8.6-asm.8/bin directory.

6. Apply the configuration to the cluster:

   bin/istioctl install --context="${CTX_CLUSTER1}" -f cluster.yaml
   

Setting the default network

1. Set the default network on the istio-system namespace.

   In the following example, NETWORK_ID can be any string that identifies a cluster's network. In this on-premises configuration, every cluster is on its own network, so each cluster should have a different value. NETWORK_ID has the same string limitations as a Kubernetes label described in Syntax and character set.

    kubectl --context="${CTX_CLUSTER1}" label \
   namespace istio-system topology.istio.io/network=NETWORK_ID

Configuring the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

1. Copy the following YAML to a file called istiod-service.yaml:

   cat <<EOF > istiod-service.yaml
   apiVersion: v1
   kind: Service
   metadata:
    name: istiod
    namespace: istio-system
    labels:
      istio.io/rev: asm-186-8
      app: istiod
      istio: pilot
      release: istio
   spec:
    ports:
      - port: 15010
        name: grpc-xds # plaintext
        protocol: TCP
      - port: 15012
        name: https-dns # mTLS with k8s-signed cert
        protocol: TCP
      - port: 443
        name: https-webhook # validation and injection
        targetPort: 15017
        protocol: TCP
      - port: 15014
        name: http-monitoring # prometheus stats
        protocol: TCP
    selector:
      app: istiod
      istio.io/rev: asm-186-8
   EOF
   

2. Configure the validating webhook so that it can locate the istiod service with the revision label:

   kubectl --context="${CTX_CLUSTER1}" apply -f istiod-service.yaml
   

   This command creates a service entry that lets the validating webhook automatically check configurations before they are applied.

Auto mutual TLS (auto mTLS) is enabled by default. With auto mTLS, a client sidecar proxy automatically detects if the server has a sidecar. The client sidecar sends mTLS to workloads with sidecars and sends plain text traffic to workloads without sidecars.

Caution: To secure your service mesh, we recommend that you migrate your services to mTLS STRICT mode, as described in Configuring transport security.

Checking the control plane components

Check that the control plane pods in istio-system are running:

kubectl --context="${CTX_CLUSTER1}" get pod -n istio-system

Expected output is similar to the following:

NAME                                      READY   STATUS      RESTARTS   AGE
istio-ingressgateway-74cc894bfd-786rg     1/1     Running     0          7m19s
istiod-78cdbbbdb-d7tps                    1/1     Running     0          7m36s
promsd-576b8db4d6-lqf64                   2/2     Running     1          7m19s

Injecting sidecar proxies

Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.

Your installation isn't complete until you enable automatic sidecar proxy injection (auto-injection) and restart the Pods for any workloads that were running on your cluster before you installed Anthos Service Mesh.

To enable auto-injection, you label your namespaces with the revision label that was set on istiod when you installed Anthos Service Mesh. The revision label is used by the sidecar injector webhook to associate injected sidecars with a particular istiod revision. After adding the label, any existing Pods in the namespace must be restarted for sidecars to be injected.

Before you deploy new workloads in a new namespace, make sure to configure auto-injection so that Anthos Service Mesh can monitor and secure traffic.

To enable auto-injection:

1. Use the following command to locate the revision label on istiod:

   kubectl -n istio-system get pods -l app=istiod --show-labels
   

   The output looks similar to the following:

   NAME                                READY   STATUS    RESTARTS   AGE   LABELS
   istiod-asm-186-8-5788d57586-bljj4   1/1     Running   0          23h   app=istiod,istio.io/rev=asm-186-8,istio=istiod,pod-template-hash=5788d57586
   istiod-asm-186-8-5788d57586-vsklm   1/1     Running   1          23h   app=istiod,istio.io/rev=asm-186-8,istio=istiod,pod-template-hash=5788d57586

   In the output, under the LABELS column, note the value of the istiod revision label, which follows the prefix istio.io/rev=. In this example, the value is asm-186-8.

2. Apply the revision label and remove the istio-injection label if it exists. In the following command, NAMESPACE is the name of the namespace where you want to enable auto-injection, and REVISION is the revision label you noted in the previous step.

   kubectl label namespace NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
   

   You can ignore the message "istio-injection not found" in the output. That means that the namespace didn't previously have the istio-injection label, which you should expect in new installations of Anthos Service Mesh or new deployments. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

3. If workloads were running on your cluster before you installed Anthos Service Mesh, restart the Pods to trigger re-injection.

   How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.

   You can use kubectl to perform a rolling restart:

   kubectl rollout restart deployment -n NAMESPACE
   

4. Verify that your Pods are configured to point to the new version of istiod.

   kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
   

What's next?

If you plan to use multiple clusters for your on-prem service mesh, see Installing Anthos Service Mesh on premises on multiple clusters and networks.

Otherwise, your next step is Configuring external IP addresses.