Installing Anthos Service Mesh on an existing cluster

This guide explains how to do a clean installation of Anthos Service Mesh version 1.5.10 on an existing Google Cloud GKE cluster. If you have previous version of Anthos Service Mesh installed, refer to Upgrading Anthos Service Mesh on GKE.

The installation enables the following features:

• Mesh telemetry.
• Mesh security, including Anthos Service Mesh certificate authority (Mesh CA).
• The Supported default features listed on the Supported features page.

This guide also explains how to register your cluster in the fleet that is in the same Google Cloud project as the cluster. A fleet lets you organize clusters to make multi-cluster management easier. By registering your clusters in a fleet, you can group services and other infrastructure as needed to apply consistent policies.

Before you start the installation:

• Set up your environment. Note that the installation requires kpt.
• Review the following requirements and restrictions.

Requirements

• You must have an GKE Enterprise trial license or subscription. See the GKE Enterprise Pricing guide for details.

• Your GKE cluster must meet the following requirements:

  • At least four nodes. If you need to add nodes, see Resizing a cluster.

  • The minimum machine type is e2-standard-4, which has four vCPUs. If the machine type for your cluster doesn't have at least four vCPUs, change the machine type as described in Migrating workloads to different machine types.

  • The cluster must be enrolled in a release channel. Anthos Service Mesh does not support static versions. Follow the instructions in Enrolling an existing cluster in a release channel if you have a static GKE version.

• To be included in the service mesh, service ports must be named, and the name must include the port's protocol in the following syntax: name: protocol[-suffix] where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.

• If you are installing Anthos Service Mesh on a private cluster, you must add a firewall rule to open port 15017 if you want to use automatic sidecar injection. If you don't add the firewall rule and automatic sidecar injection is enabled, you get an error when you deploy workloads. For details on adding a firewall rule, see Adding firewall rules for specific use cases.

• If you have created a service perimeter in your organization, you might need to add the Mesh CA service to the perimeter. See Adding Mesh CA to a service perimeter for more information.

Restrictions

Only one installation of Anthos Service Mesh per Google Cloud project is supported. Multiple mesh deployments in a single project aren't supported.

Certificate data

Certificates from Mesh CA include the following data about your application's services:

• The Google Cloud project ID
• The GKE namespace
• The GKE service account name

Important: The certificates issued by Mesh CA should only be used to enable secure service-to-service communication within your service mesh, and not be used for any other purpose. These certificates are sent whenever services attempt to communicate with each other using mutual TLS. Make sure that you don't inadvertently expose confidential information by using these certificates when communicating outside your service mesh.

Setting up your project

1. Get the project ID of the project that the cluster was created in:

   gcloud

   gcloud projects list

   Console

   1. In the Google Cloud console, go to the Dashboard page:

      Go to the Dashboard page

   2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select your project.

      The project ID is displayed on the project Dashboard Project info card.

2. Create an environment variable for the project ID:

   export PROJECT_ID=YOUR_PROJECT_ID
   

3. Set the default project ID for the gcloud command-line tool:

   gcloud config set project ${PROJECT_ID}

4. Create an environment variable for the project number:

   export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")

   Important: Many of the commands in this guide depend on these environment variables. If you disconnect from your shell environment and reconnect later, you need to set up the variables again. Additionally, the commands assume that you've set the default project for the gcloud command-line tool.

5. Set the required Identity and Access Management (IAM) roles. If you are a Project Owner, you have all the necessary permissions to complete the installation and register your cluster with your environ. If you aren't a Project Owner, you need someone who is to grant you the following specific IAM roles. In the following command, change GCP_EMAIL_ADDRESS to the account that you use to log in to Google Cloud.

   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member user:GCP_EMAIL_ADDRESS \
        --role=roles/editor \
        --role=roles/compute.admin \
        --role=roles/container.admin \
        --role=roles/resourcemanager.projectIamAdmin \
        --role=roles/iam.serviceAccountAdmin \
        --role=roles/iam.serviceAccountKeyAdmin \
        --role=roles/gkehub.admin

   To learn more about how to grant IAM roles, refer to Granting, changing, and revoking access to resources. For a description of these roles, see Permissions required to install Anthos Service Mesh

6. Enable the following APIs:

   gcloud services enable \
       container.googleapis.com \
       compute.googleapis.com \
       monitoring.googleapis.com \
       logging.googleapis.com \
       cloudtrace.googleapis.com \
       meshca.googleapis.com \
       meshtelemetry.googleapis.com \
       meshconfig.googleapis.com \
       iamcredentials.googleapis.com \
       anthos.googleapis.com \
       gkeconnect.googleapis.com \
       gkehub.googleapis.com \
       cloudresourcemanager.googleapis.com

   Enabling the APIs can take a minute or more to complete. When the APIs are enabled, you see output similar to the following:

   Operation "operations/acf.601db672-88e6-4f98-8ceb-aa3b5725533c" finished
   successfully.
   

Setting up an existing GKE cluster

This section explains how to set up an existing GKE cluster with the options that are required for Anthos Service Mesh. For more information, see the GKE documentation.

1. Create the following environment variables:

   • Set the cluster name.

     export CLUSTER_NAME=YOUR_CLUSTER_NAME

   • Set the CLUSTER_LOCATION to either your cluster zone or cluster region.

     export CLUSTER_LOCATION=YOUR_ZONE_OR_REGION

   • Set the workload pool.

     export WORKLOAD_POOL=${PROJECT_ID}.svc.id.goog

   • Set the mesh ID.

     export MESH_ID="proj-${PROJECT_NUMBER}"

2. Set the default zone or region for the Google Cloud CLI.

   • If you have a single-zone cluster, set the default zone:

     gcloud config set compute/zone ${CLUSTER_LOCATION}

   • If you have a regional cluster, set the default region:

     gcloud config set compute/region ${CLUSTER_LOCATION}

   Tip: To make setting up your shell environment easier in the future, you can copy and paste the export statements for each environment variable to a simple shell script that you source when you start a new shell. You can also add the gcloud commands that set default values to the script. Or you can use gcloud init to create and activate a named gcloud configuration.

3. Set the mesh_id label on the cluster, which is required for metrics to get displayed on the Anthos Service Mesh dashboard in the Google Cloud console. If your cluster has existing labels that you want to keep, you must include those labels when adding the mesh_id label.

   1. To see if your cluster has existing labels:

      gcloud container clusters describe ${CLUSTER_NAME}

      Look for the resourceLabels field in the output. Each label is stored on a separate line under the resourceLabels field, for example:

      resourceLabels:
        csm: ''
        env: dev
        release: stable

      If the csm label is in the output, you don't need to preserve it. The mesh_id label replaces the csm label.

      For convenience, you can add the labels to an environment variable. In the following, replace YOUR_EXISTING_LABELS with a comma-separated list of the existing labels on your cluster in the format KEY=VALUE, for example: env=dev,release=stable

      export EXISTING_LABELS="YOUR_EXISTING_LABELS"

   2. Set the mesh_id label:

      • If your cluster has existing labels that you want to keep, update the cluster with the mesh_id and the existing labels:

        gcloud container clusters update ${CLUSTER_NAME} \
          --update-labels=mesh_id=${MESH_ID},${EXISTING_LABELS}

      • If you cluster doesn't have any existing labels, update the cluster with only the mesh_id label:

        gcloud container clusters update ${CLUSTER_NAME} \
          --update-labels=mesh_id=${MESH_ID}

4. Enable Workload Identity:

   gcloud container clusters update ${CLUSTER_NAME} \
      --workload-pool=${WORKLOAD_POOL}

5. Enable Cloud Monitoring and Cloud Logging on GKE:

   gcloud container clusters update ${CLUSTER_NAME} \
      --enable-stackdriver-kubernetes

6. If your cluster is using a static version of GKE, enroll the cluster in a release channel.

Setting credentials and permissions

Before continuing, make sure that you enabled all the required APIs. If you have any doubt, it doesn't hurt to run the gcloud services enable command again.

1. Initialize your project to ready it for installation. Among other things, this command creates a service account to let Istio components, such as the sidecar proxy, securely access your project's data and resources:

   curl --request POST \
     --header "Authorization: Bearer $(gcloud auth print-access-token)" \
     --data '' \
     https://meshconfig.googleapis.com/v1alpha1/projects/${PROJECT_ID}:initialize

   The command responds with empty curly braces: {}

   If you install a new version of Anthos Service Mesh on this cluster in the future, you don't need to re-run the command, but running the command again doesn't affect your installation.

2. Get authentication credentials to interact with the cluster:

   gcloud container clusters get-credentials ${CLUSTER_NAME}

3. Grant cluster admin permissions to the current user. You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

   kubectl create clusterrolebinding cluster-admin-binding \
     --clusterrole=cluster-admin \
     --user="$(gcloud config get-value core/account)"

   If you see the "cluster-admin-binding" already exists error, you can safely ignore it and continue with the existing cluster-admin-binding.

Registering your cluster

You must register your cluster with the project's fleet to gain access to the unified user interface in the Google Cloud console. A fleet provides a unified way to view and manage the clusters and their workloads, including clusters outside Google Cloud.

Create a Google Cloud service account and key file

A JSON file containing service account credentials is required to register a cluster. To follow the principle of least privilege, we recommend that you create a distinct service account for each cluster that you register.

To create a service account and key file:

1. Select a name for the service account and create an environment variable for it:

   export SERVICE_ACCOUNT_NAME=SERVICE_ACCOUNT_NAME
   

2. Create the service account:

   gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME}

3. List all of a project's service accounts to confirm the service account was created:

   gcloud iam service-accounts list

4. Bind the gkehub.connect IAM role to the service account:

   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
      --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
      --role="roles/gkehub.connect"

5. Create an environment variable for the local filepath where you want to save the JSON file. We recommend that you name the file using the service account name and your project ID, such as: /tmp/creds/${SERVICE_ACCOUNT_NAME}-${PROJECT_ID}.json

   export SERVICE_ACCOUNT_KEY_PATH=LOCAL_KEY_PATH

6. Download the service account's private key JSON file:

   gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_PATH} \
      --iam-account=${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

Register the cluster

In the following command, replace MEMBERSHIP_NAME with a name that uniquely represents the cluster being registered on the Hub.

gcloud container hub memberships register MEMBERSHIP_NAME \
    --gke-cluster=${CLUSTER_LOCATION}/${CLUSTER_NAME} \
    --service-account-key-file=${SERVICE_ACCOUNT_KEY_PATH}

The command responds with output similar to the following:

kubeconfig entry generated for CLUSTER_NAME.
Waiting for membership to be created...done.
Created a new membership [projects/PROJECT_ID/locations/global/memberships/MEMBERSHIP_NAME] for the cluster [MEMBERSHIP_NAME]
Generating the Connect Agent manifest...
Deploying the Connect Agent on cluster [MEMBERSHIP_NAME] in namespace [gke-connect]...
Deployed the Connect Agent on cluster [MEMBERSHIP_NAME] in namespace [gke-connect].
Finished registering the cluster [MEMBERSHIP_NAME] with the Hub.

This service account key is stored as a secret named creds-gcp in the gke-connect namespace.

For more information about cluster registration, see Registering a cluster in the Connect documentation.

Downloading the installation file

Before continuing, verify that the ASM Mesh Data Plane Service Account is a member of the project:

gcloud projects get-iam-policy ${PROJECT_ID} | grep -B 1 'roles/meshdataplane.serviceAgent'

If the previous command doesn't output anything, go back to the Set credentials and permissions section and run the curl command.

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-linux.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-linux.tar.gz.1.sig
openssl dgst -verify - -signature istio-1.5.10-asm.2-linux.tar.gz.1.sig istio-1.5.10-asm.2-linux.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-linux.tar.gz

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.5.10-asm.2-osx.tar.gz.1.sig istio-1.5.10-asm.2-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.5.10-asm.2-win.zip.1.sig
openssl dgst -verify - -signature istio-1.5.10-asm.2-win.zip.1.sig istio-1.5.10-asm.2-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.5.10-asm.2-win.zip

The command creates an installation directory in your current working directory named istio-1.5.10-asm.2 that contains:

• Sample applications in samples
• The following tools in the bin directory:
  • istioctl: You use istioctl to install Anthos Service Mesh.
  • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh. (Currently, asmctl isn't supported on GKE on VMware.)

1. Ensure that you're in the Anthos Service Mesh installation's root directory.

   cd istio-1.5.10-asm.2

2. For convenience, add the tools in the /bin directory to your PATH:

   export PATH=$PWD/bin:$PATH

Preparing resource configuration files

When you run the istioctl apply command to install Anthos Service Mesh, you specify -f istio-operator.yaml on the command line. This file contains information about your project and cluster that is needed to enable the Mesh telemetry and Mesh security features. You need to download the istio-operator.yaml and other resource configuration files and set the project and cluster information.

To prepare the resource configuration files:

1. If you haven't already, install kpt:

   gcloud components install kpt
   

2. Optionally, create a new directory for the Anthos Service Mesh package resource configuration files. If you plan to set up more than one cluster, you might want to use the cluster name as the directory name.

3. Change to the directory where you want to download the Anthos Service Mesh package.

4. Download the Anthos Service Mesh package to the current working directory:

   kpt pkg get \
   https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git/asm@release-1.5-asm .
   

5. Set the cluster name:

     kpt cfg set asm gcloud.container.cluster ${CLUSTER_NAME}

6. Optionally, customize the resource configuration files by using the kpt setters. By default, these setters use the defaults for gcloud config. If you set the gcloud config defaults, or if you want to change the values, run the following setters:

   • Set the project ID:

     kpt cfg set asm gcloud.core.project ${PROJECT_ID}

   • Set the default zone or region:

     kpt cfg set asm gcloud.compute.location ${CLUSTER_LOCATION}

7. Optionally, you can check in the resource configuration files to your own source control system, such as Cloud Source Repositories, so that you can track changes to the files.

Installing Anthos Service Mesh

This section explains how to install Anthos Service Mesh and enable:

• The Supported default features listed on the Supported features page.
• Anthos Service Mesh certificate authority (Mesh CA).
• The telemetry data pipeline that powers the Anthos Service Mesh dashboards in the Google Cloud console.

For information on enabling the Supported optional features, see Enabling optional features.

To install Anthos Service Mesh:

Choose one of the following commands to configure Anthos Service Mesh in PERMISSIVE mutual TLS (mTLS) authentication mode or STRICT mTLS mode.

PERMISSIVE mTLS

istioctl manifest apply --set profile=asm \
  -f asm/cluster/istio-operator.yaml

Caution: In PERMISSIVE mode, your services can accept both plaintext traffic and mutual TLS traffic at the same time and send plaintext intra-mesh traffic by default. To secure your service mesh after the installation, migrate your services to mutual TLS STRICT mode, as described in Mutual TLS Migration.

STRICT mTLS

istioctl manifest apply --set profile=asm \
  -f asm/cluster/istio-operator.yaml \
  --set values.global.mtls.enabled=true

Check the control plane components

Check that the control plane pods in istio-system are up:

kubectl get pod -n istio-system

Expected output is similar to the following:

NAME                                      READY   STATUS      RESTARTS   AGE
istio-ingressgateway-74cc894bfd-786rg     1/1     Running     0          7m19s
istiod-78cdbbbdb-d7tps                    1/1     Running     0          7m36s
promsd-576b8db4d6-lqf64                   2/2     Running     1          7m19s

Validating the installation

Beta

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

We recommend that you use the asmctl analysis tool to validate the basic configuration of your project, cluster, and workloads. If an asmctl test fails, asmctl recommends solutions, if possible. The asmctl validate command runs basic tests that check:

1. That the APIs required by Anthos Service Mesh are enabled on the project.
2. That the Istio-Ingressgateway is properly configured to call Mesh CA.
3. The general health of Istiod and Istio-Ingressgateway.

If you run the asmctl validate command with the optional --with-testing-workloads flag, in addition to the basic tests, asmctl runs security tests that check:

1. Mutual TLS (mTLS) communication is configured properly.
2. Mesh CA can issue certificates.

To run the security tests, asmctl deploys workloads on your cluster in a test namespace, runs the mTLS communication tests, outputs the results, and deletes the test namespace.

To run asmctl:

1. Ensure that gcloud application-default credentials are set:

    gcloud auth application-default login
   

2. If you haven't already, get authentication credentials to interact with the cluster:

    gcloud container clusters get-credentials ${CLUSTER_NAME}
   

3. To run both the basic and security tests (assuming istio-1.5.10-asm.2/bin) is in yourPATH):

   asmctl validate --with-testing-workloads
   

   On success, the command responds with output similar to the following:

   [asmctl version 0.3.0]
   Using Kubernetes context: example-project_us-central1-example-cluster
   To change the context, use the --context flag
   Validating enabled APIs
   OK
   Validating ingressgateway configuration
   OK
   Validating istio system
   OK
   Validating sample traffic
   Launching example services...
   Sent traffic to example service http code: 200
   verified mTLS configuration
   OK
   Validating issued certs
   OK
   

Injecting sidecar proxies

Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.

Any workloads that were running on your cluster before you installed Anthos Service Mesh need to have the sidecar proxy injected or updated so they have the current Anthos Service Mesh version. Before you deploy new workloads, make sure to configure sidecar proxy injection so that Anthos Service Mesh can monitor and secure traffic.

You can enable automatic sidecar injection with one command, for example:

kubectl label namespace NAMESPACE istio-injection=enabled --overwrite

where NAMESPACE is the name of the namespace for your application's services or default if you didn't explicitly create a namespace.

For more information, see Injecting sidecar proxies.

Viewing the Anthos Service Mesh dashboards

After you have workloads deployed on your cluster with the sidecar proxies injected, you can explore the Anthos Service Mesh pages in the Google Cloud console to see all of the observability features that Anthos Service Mesh offers. Note that it takes about one or two minutes for telemetry data to be displayed in the Google Cloud console after you deploy workloads.

Access to Anthos Service Mesh in the Google Cloud console is controlled by Identity and Access Management (IAM). To access the Anthos Service Mesh pages, a Project Owner must grant users the Project Editor or Viewer role, or the more restrictive roles described in Controlling access to Anthos Service Mesh in the Google Cloud console.

1. In the Google Cloud console, go to Anthos Service Mesh.

   Go to Anthos Service Mesh

2. Select the Google Cloud project from the drop-down list on the menu bar.

3. If you have more than one service mesh, select the mesh from the Service Mesh drop-down list.

To learn more, see Exploring Anthos Service Mesh in the Google Cloud console.

In addition to the Anthos Service Mesh pages, metrics related to your services (such as the number of requests received by a particular service) are sent to Cloud Monitoring, where they appear in the Metrics Explorer.

To view metrics:

1. In the Google Cloud console, go to the Monitoring page:

   Go to Monitoring

2. Select Resources > Metrics Explorer.

For a full list of metrics, see Istio metrics in the Cloud Monitoring documentation.

What's next

• Deploy a sample application