Installing Anthos Service Mesh on an existing cluster

Anthos Service Mesh, is an Istio-compatible framework for connecting, monitoring, and securing services running on Google Kubernetes Engine (GKE) and Anthos GKE on-prem. It lets you create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, without requiring any changes in service code. Anthos Service Mesh automatically injects a sidecar proxy for each of your applications' pods. The sidecar proxy intercepts all network traffic to and from the pods. Anthos Service Mesh also configures an ingress gateway to manage inbound traffic to the mesh. You can use open source Istio APIs to configure policies that are enforced on sidecars and gateways.

This guide explains how to install Anthos Service Mesh version 1.4.6-asm.0 on an existing Google Cloud GKE cluster. If open source Istio 1.4.x or a previous version of Anthos Service Mesh is installed on the cluster, you can follow this guide to upgrade to Anthos Service Mesh 1.4.6-asm.0. If you have an earlier version of open source Istio installed, first upgrade to Istio 1.4, and then follow the steps in this guide.

The installation enables the following features:

Note: When you install Anthos Service Mesh, the telemetry pipeline that powers the Anthos Service Mesh dashboard in the Google Cloud Console is installed automatically. However, the Anthos Service Mesh dashboard itself is in beta.

Before you begin

Requirements

  • You must have an Anthos trial license and subscription. See the Anthos Pricing guide for details.

  • Make sure your cluster master version is listed in Supported environments. To check your cluster version:

    kubectl version | grep Server
    

    Output like the following is displayed:

    Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", ...}
    

    If you need to upgrade your cluster to a supported version, see Manually upgrading a cluster or node pool.

  • Your GKE cluster must meet the following requirements:

  • Review Requirements for Pods and Services before you deploy workloads.

  • If you are installing Anthos Service Mesh on a private cluster, you must add a firewall rule to open port 9443 if you want to use automatic sidecar injection. If you don't add the firewall rule and automatic sidecar injection is enabled, you get an error when you deploy workloads. For details on adding a firewall rule, see Adding firewall rules for specific use cases.

  • Your cluster must be registered to an Anthos Environ using Connect for Anthos. Your project's Environ provides a unified way to view and manage your clusters and their workloads as part of Anthos, including clusters outside Google Cloud. Anthos charges apply only to your registered clusters. You can find out how to register a cluster in Registering a cluster.

Restrictions

Only one installation of Anthos Service Mesh per Google Cloud project is supported. Multiple mesh deployments in a single project aren't supported.

Certificate data

Certificates from Mesh CA include the following data about your application's services:

  • The Google Cloud project ID
  • The GKE namespace
  • The GKE service account name

Setting up your environment

You can follow the installation guides using Cloud Shell, an in-browser command line interface to your Google Cloud resources, or your own computer running Linux or macOS.

Option A: Use Cloud Shell

Cloud Shell provisions a g1-small Compute Engine virtual machine (VM) running a Debian-based Linux operating system. The advantages to using Cloud Shell are:

  • Cloud Shell includes the gcloud, kubectl and helm command-line tools that you need.

  • Your Cloud Shell $HOME directory has 5GB persistent storage space.

  • You have your choice of text editors:

    • Code editor, which you access by clicking edit at the top of the Cloud Shell window.

    • Emacs, Vim, or Nano, which you access from the command line in Cloud Shell.

To use Cloud Shell:

  1. Go to the Cloud Console.
  2. Select your Cloud project.
  3. Click the Activate Cloud Shell button at the top of the Cloud Console window.

    Google Cloud Platform console

    A Cloud Shell session opens inside a new frame at the bottom of the Cloud Console and displays a command-line prompt.

    Cloud Shell session

Option B: Use command-line tools locally

On your local machine, install the following tools if you don't already have them:

  1. Install and initialize the Cloud SDK (the gcloud command-line tool).

    If you already have the Cloud SDK installed, make sure to update the components:

    gcloud components update
    
  2. Install kubectl:

    gcloud components install kubectl
    

Setting up your project

  1. Authenticate with the Cloud SDK:
    gcloud auth login
  2. Get your Cloud project ID and create an environment variable for it:
    export PROJECT_ID=YOUR_PROJECT_ID
  3. Set the default project ID for the gcloud command-line tool:
    gcloud config set project ${PROJECT_ID}
  4. Create an environment variable for the project number:
    export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
  5. Set the required Cloud Identity and Access Management (Cloud IAM) roles. You need the following roles on the Cloud project:
    Role name Role ID Description
    Project Editor roles/editor Permissions for actions that modify state, such as changing existing resources.
    Kubernetes Engine Admin roles/container.admin Provides access to full management of Container Clusters and their Kubernetes API objects.
    Project IAM Admin roles/resourcemanager.projectIamAdmin Provides permissions to administer Cloud IAM policies on projects.

    If you are a Project Owner, you don't need to add those roles because the Project Owner role has all the necessary permissions. For details on setting the roles, see Granting, changing, and revoking access to resources.

  6. Enable the following APIs:
    gcloud services enable \
        container.googleapis.com \
        compute.googleapis.com \
        stackdriver.googleapis.com \
        meshca.googleapis.com \
        meshtelemetry.googleapis.com \
        meshconfig.googleapis.com \
        iamcredentials.googleapis.com \
        anthos.googleapis.com
    

    Enabling the APIs can take a minute or more to complete.

Setting up an existing GKE cluster

This section explains how to set up an existing GKE cluster with the options that are required for Anthos Service Mesh. For more information, see the GKE documentation.

  1. Create the following environment variables:

    export CLUSTER_NAME=YOUR_CLUSTER_NAME
    
    export CLUSTER_ZONE=YOUR_CLUSTER_ZONE
    
    export IDNS=${PROJECT_ID}.svc.id.goog
    
    export MESH_ID="proj-${PROJECT_NUMBER}"
  2. Set the default zone for the gcloud command-line tool:

    gcloud config set compute/zone ${CLUSTER_ZONE}
    
  3. See if your cluster has existing labels:

    gcloud container clusters describe ${CLUSTER_NAME}
    

    Look for the resourceLabels field in the output. Each label is stored on a separate line under the resourceLabels field. If you don't see resourceLabels in the output, then your cluster doesn't have any labels. For convenience, you can add them to an environment variable called EXISTING_LABELS. For example:

    export EXISTING_LABELS="env=dev,release=stable"
    

    Tip: To make setting up your shell environment easier in the future, you can copy and paste the export statements for each environment variable to a simple shell script that you source when you start a new shell. You can also add the gcloud commands that set the default zone and project to the script. Or you can use gcloud config configurations to create and activate a named gcloud configuration.

  4. Update the cluster:

    • If your cluster has existing labels that you want to keep, you must include those labels when adding the mesh_id label:

      gcloud beta container clusters update ${CLUSTER_NAME} --identity-namespace=${IDNS}
      gcloud beta container clusters update ${CLUSTER_NAME} --enable-stackdriver-kubernetes
      gcloud beta container clusters update ${CLUSTER_NAME} --update-labels mesh_id=${MESH_ID},${EXISTING_LABELS}
    • If you cluster doesn't have any existing labels, update the cluster with the mesh_id label:

      gcloud beta container clusters update ${CLUSTER_NAME} --identity-namespace=${IDNS}
      gcloud beta container clusters update ${CLUSTER_NAME} --enable-stackdriver-kubernetes
      gcloud beta container clusters update ${CLUSTER_NAME} --update-labels mesh_id=${MESH_ID}

    The clusters update command requires the following options:

    • enable-stackdriver-kubernetes: Enables Cloud Monitoring and Cloud Logging on GKE.

    • identity-namespace=${IDNS}: Enables Workload Identity, which is the recommended way to safely access Google Cloud services from GKE applications.

    • update-labels mesh_id=${MESH_ID}: Sets the mesh_id label on the cluster, which is required for metrics to get displayed on the Anthos Service Mesh Dashboard in the Cloud Console.

Preparing to install Anthos Service Mesh

Before installing Anthos Service Mesh, you need to:

  • Set required credentials and permissions.
  • Download and extract the Anthos Service Mesh installation file.

Set credentials and permissions

  1. Initialize your project to ready it for installation. Among other things, this command creates a service account to let Istio components, such as the sidecar proxy, securely access your project's data and resources:
    curl --request POST \
      --header "Authorization: Bearer $(gcloud auth print-access-token)" \
      --data '' \
      https://meshconfig.googleapis.com/v1alpha1/projects/${PROJECT_ID}:initialize

    The command responds with empty curly braces: {}

    If you install a new version of Anthos Service Mesh on this cluster in the future, you don't need to re-run the command, but running the command again doesn't affect your installation.

  2. Get authentication credentials to interact with the cluster:
    gcloud container clusters get-credentials ${CLUSTER_NAME}

    This command configures kubectl to use the specified cluster.

  3. Grant cluster admin permissions to the current user. You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:
    kubectl create clusterrolebinding cluster-admin-binding \
      --clusterrole=cluster-admin \
      --user="$(gcloud config get-value core/account)"

    If you see the "cluster-admin-binding" already exists error, you can safely ignore it and continue with the existing cluster-admin-binding.

Download the installation file

Linux

  1. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-linux.tar.gz
  2. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-linux.tar.gz.1.sig
    openssl dgst -verify - -signature istio-1.4.6-asm.0-linux.tar.gz.1.sig istio-1.4.6-asm.0-linux.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.4.6-asm.0-linux.tar.gz

    The command creates an installation directory in your current working directory named istio-1.4.6-asm.0 that contains:

    • Sample applications in samples
    • The following tools in the bin directory:
      • istioctl: You use istioctl to install Anthos Service Mesh.
      • asmctl: You use asmctl to help validate your security configuration after installing Anthos Service Mesh.

  4. Ensure that you're in the Anthos Service Mesh installation's root directory.
    cd istio-1.4.6-asm.0
  5. For convenience, add the tools in the /bin directory to your PATH:
    export PATH=$PWD/bin:$PATH

Mac OS

  1. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-osx.tar.gz
  2. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-osx.tar.gz.1.sig
    openssl dgst -verify - -signature istio-1.4.6-asm.0-osx.tar.gz.1.sig istio-1.4.6-asm.0-osx.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
    tar xzf istio-1.4.6-asm.0-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.4.6-asm.0 that contains:

    • Sample applications in samples
    • The istioctl client binary in the bin directory. You can use istioctl to manually inject Envoy as a sidecar proxy and to create routing rules and policies.

  4. Ensure that you're in the Anthos Service Mesh installation's root directory.
    cd istio-1.4.6-asm.0
  5. For convenience, add the istioctl client to your PATH:
    export PATH=$PWD/bin:$PATH

Windows

  1. Download the Anthos Service Mesh installation file to your current working directory:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-win.zip
  2. Download the signature file and use openssl to verify the signature:
    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.4.6-asm.0-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.4.6-asm.0-win.zip.1.sig istio-1.4.6-asm.0-win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK

  3. Extract the contents of the file to any location on your file system. There is a directory named istio-1.4.6-asm.0 that contains:
    • Sample applications in samples
    • The istioctl client binary in the bin directory. You can use istioctl to manually inject Envoy as a sidecar proxy and to create routing rules and policies.

  4. Ensure that you're in the Anthos Service Mesh installation's root directory.
    cd istio-1.4.6-asm.0
  5. For convenience, add the istioctl client to your PATH.

Installing Anthos Service Mesh

This section explains how to install Anthos Service Mesh and enable:

  • The Supported default features listed on the Supported features page.
  • Anthos Service Mesh certificate authority (Mesh CA).
  • The telemetry data pipeline that powers the Anthos Service Mesh dashboard in the Google Cloud Console.

For information on enabling the Supported optional features, see Enabling optional features.

To install Anthos Service Mesh:

Choose one of the following commands to install Anthos Service Mesh in PERMISSIVE mutual TLS (mTLS) authentication mode or STRICT mTLS mode.

PERMISSIVE mTLS

istioctl manifest apply --set profile=asm \
  --set values.global.trustDomain=${IDNS} \
  --set values.global.sds.token.aud=${IDNS} \
  --set values.nodeagent.env.GKE_CLUSTER_URL=https://container.googleapis.com/v1/projects/${PROJECT_ID}/locations/${CLUSTER_ZONE}/clusters/${CLUSTER_NAME} \
  --set values.global.meshID=${MESH_ID} \
  --set values.global.proxy.env.GCP_METADATA="${PROJECT_ID}|${PROJECT_NUMBER}|${CLUSTER_NAME}|${CLUSTER_ZONE}"

STRICT mTLS

istioctl manifest apply --set profile=asm \
  --set values.global.trustDomain=${IDNS} \
  --set values.global.sds.token.aud=${IDNS} \
  --set values.nodeagent.env.GKE_CLUSTER_URL=https://container.googleapis.com/v1/projects/${PROJECT_ID}/locations/${CLUSTER_ZONE}/clusters/${CLUSTER_NAME} \
  --set values.global.meshID=${MESH_ID} \
  --set values.global.proxy.env.GCP_METADATA="${PROJECT_ID}|${PROJECT_NUMBER}|${CLUSTER_NAME}|${CLUSTER_ZONE}" \
  --set values.global.mtls.enabled=true

If you are upgrading from open source Istio 1.4.x or an earlier version of Anthos Service Mesh, the upgrade requires redeploying the control plane components, and the installation takes about 5 to 10 minutes to complete. The old control plane components are terminated and then deleted as the new components are installed. You can check the progress by looking at the value in the AGE column of the workloads.

kubectl get pod -n istio-system

Example output:

NAME                                     READY   STATUS        RESTARTS   AGE
istio-galley-76d684bf9-jwz65             2/2     Running       0          5m36s
istio-ingressgateway-5bfdf7c586-v6wxx    2/2     Terminating   0          25m
istio-ingressgateway-7b598c5557-b88md    2/2     Running       0          5m44s
istio-nodeagent-bnjg7                    1/1     Running       0          5m16s
istio-nodeagent-cps2j                    1/1     Running       0          5m10s
istio-nodeagent-f4x46                    1/1     Running       0          5m26s
istio-nodeagent-jbl5x                    1/1     Running       0          5m38s
istio-pilot-5dc4bc4dbf-ds5dh             2/2     Running       0          5m30s
istio-pilot-74665549c5-7r6qh             2/2     Terminating   0          25m
istio-sidecar-injector-7ddff4b99-b76l7   1/1     Running       0          5m36s
promsd-6d4d5b7c5c-dgnd7                  2/2     Running       0          5m30s

In this example, there are two instances of istio-ingressgateway and istio-pilot. The instances with 25min the AGE column are being terminated. All the other components are newly deployed from the ASM 1.4.6-asm.0 package.

Wait for the deployment to finish:

kubectl wait --for=condition=available --timeout=600s deployment --all -n istio-system

Output:

deployment.extensions/istio-galley condition met
deployment.extensions/istio-ingressgateway condition met
deployment.extensions/istio-pilot condition met
deployment.extensions/istio-sidecar-injector condition met
deployment.extensions/promsd condition met

Verifying the installation

Check that the control plane Pods in istio-system are up:

kubectl get pod -n istio-system

Expect to see output similar to the following:

NAME                                      READY   STATUS      RESTARTS   AGE
istio-galley-5c65896ff7-m2pls             2/2     Running     0          18m
istio-ingressgateway-587cd459f-q6hqt      2/2     Running     0          18m
istio-nodeagent-74w69                     1/1     Running     0          18m
istio-nodeagent-7524w                     1/1     Running     0          18m
istio-nodeagent-7652w                     1/1     Running     0          18m
istio-nodeagent-7948w                     1/1     Running     0          18m
istio-pilot-9db77b99f-7wfb6               2/2     Running     0          18m
istio-sidecar-injector-69c4d9f875-dt8rn   1/1     Running     0          18m
promsd-55f464d964-lqs7w                   2/2     Running     0          18m

You should see an instance of the istio-nodeagent for each node in your cluster. Mesh CA, which replaces the Citadel OSS Istio component, creates the node agents to issue mTLS certificates for the workloads running in your service mesh.

Validating security features

You can use the asmctl analysis tool to help validate your security installation. asmctl comes with your Anthos Service Mesh installation and is located in the YOUR_ASM_INSTALLATION_DIRECTORY/bin directory.

asmctl checks correctness of basic configurations in your project, cluster, and workloads, and if possible, recommends solutions. asmctl can also verify that certificates were issued by Mesh CA. asmctl validates the following:

  1. Check the enabled APIs on the project for APIs that are required by Anthos Service Mesh.
  2. Check that Node-Agent is properly configured to call Mesh CA.
  3. Check the general health of Istio-Pilot and Istio-Galley.
  4. Check that Istio-Pilot uses a certificate issued by Mesh CA.

Optionally, you can run asmctl to validate mTLS communication as well as the other validation checks. To test this, asmctl deploys workloads on your cluster in a test namespace, runs the mTLS communication tests, outputs the results, and deletes the test namespace.

  1. Ensure that gcloud application-default credentials are set:

     gcloud auth application-default login
    
  2. If you have not already, get authentication credentials to interact with the cluster:

     gcloud container clusters get-credentials ${CLUSTER_NAME} --zone ${CLUSTER_ZONE} --project ${PROJECT_ID}
    
  3. To run asmctl (assuming you added YOUR_ASM_INSTALLATION_DIRECTORY/bin to your PATH):

    asmctl validate
    

    Example output:

    Using Kubernetes context: meshci145g-20200219_us-central1-a_meshci145g
    To change the context, use the --context flag
    Validating enabled APIs
    OK
    Validating Node-Agent configuration
    OK
    Validating Istio system
    OK
    Validating issued certs
    OK
  4. To validate mTLS communication in addition to the other validation tests:

    asmctl validate --with-testing-workloads
    

    Example output:

    Using Kubernetes context: meshci145g-20200219_us-central1-a_meshci145g
    To change the context, use the --context flag
    Validating enabled APIs
    OK
    Validating Node-Agent configuration
    OK
    Validating Istio system
    OK
    Validating issued certs
    OK
    Validating sample traffic
    Launching example services...
    Sent traffic to example service http code: 200
    verified mTLS configuration
    OK

Enable Pod Security Policies

For the best security on your service mesh, we recommend that you enable Pod Security Policies.

Updating your authorization policies

If you are upgrading from open source Istio 1.4.x or from an earlier version of Anthos Service Mesh and have existing authorization policies, you might need to update them. See Updating your authorization policies for more information.

Updating sidecar proxies

Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.

Any workloads that were running on your cluster before you installed Anthos Service Mesh need to have the sidecar proxy injected or updated so they have the current Anthos Service Mesh version. Before you deploy new workloads, make sure to configure sidecar proxy injection so that Anthos Service Mesh can monitor and secure traffic. You can enable automatic sidecar injection with one command, for example:

kubectl label namespace NAMESPACE istio-injection=enabled --overwrite

where NAMESPACE is the name of the namespace for your application's services or default if you didn't explicitly create a namespace.

For more information, see Injecting sidecar proxies.

Viewing the Anthos Service Mesh Dashboard

After you have workloads deployed on your cluster with the sidecar proxies injected, you can explore the Anthos Service Mesh Dashboard to see all of the observability features that Anthos Service Mesh offers.

Access to the Anthos Service Mesh Dashboard is controlled by Cloud Identity and Access Management (Cloud IAM). To access the Anthos Service Mesh Dashboard, a Project Owner must grant users the Project Editor or Viewer role, or the more restrictive roles described Controlling access to the Service Mesh Dashboard.

  1. In the Google Cloud Console, go to the Anthos Service Mesh dashboard.

    Go to the Anthos Service Mesh dashboard

  2. Select the Cloud project from the drop-down list on the menu bar.

  3. If you have more than one service mesh, select the mesh from the Service Mesh drop-down list.

To learn more, see Exploring the Anthos Service Mesh Dashboard.

In addition to the Anthos Service Mesh Dashboard, metrics related to your services (such as the number of requests received by a particular service) are sent to Cloud Monitoring, where they appear in the Metrics Explorer.

To view metrics:

  1. In the Google Cloud Console, go to the Monitoring page:

    Go to Monitoring

  2. Select Resources > Metrics Explorer.

For a full list of metrics, see Istio metrics in the Cloud Monitoring documentation.

What's next