Version 1.7

Add clusters to an Anthos Service Mesh

This guide explains how to join two clusters into a single Anthos Service Mesh using Mesh CA or Citadel, and enable cross-cluster load balancing. You can easily extend this process to incorporate any number of clusters into your mesh.

A multi-cluster Anthos Service Mesh configuration can solve several crucial enterprise scenarios, such as scale, location, and isolation. For more information, see Multi-cluster use cases. In addition, you should optimize your applications to get the most benefit from a service mesh. For more information, see Preparing an application for Anthos Service Mesh.

Prerequisites

This guide assumes that you have two or more Google Cloud GKE clusters that meet the following requirements:

  • Anthos Service Mesh version 1.6.8 or higher installed on the clusters. See Installation overview to install or upgrade your clusters to the required version.
  • If you join clusters that are not in the same project, they must be installed using the asm-gcp-multiproject profile and the clusters must be in a shared VPC configuration together on the same network. In addition, we recommend that you have one project to host the shared VPC, and two service projects for creating clusters. For more information, see Setting up clusters with Shared VPC.
  • If you use Citadel CA, use the same custom root CA for both clusters.

Setting project and cluster variables

  1. Set a working folder for convenience. This is the folder in which you downloaded and extracted the Anthos Service Mesh files in the prerequisite step, Preparing to install Anthos Service Mesh.

    export PROJECT_DIR=YOUR_WORKING_FOLDER
  2. Create a context variable for each cluster. The context is a string constructed from the cluster project IDs, cluster names, and locations. For the location values, use the cluster's location, for example us-west2-a. In this example, a mesh contains one cluster already and you're adding another cluster to the mesh:

    export CTX_1=gke_CLUSTER_1_PROJECT_ID_CLUSTER_1_LOCATION_CLUSTER_1_NAME
    export CTX_2=gke_CLUSTER_2_PROJECT_ID_CLUSTER_2_LOCATION_CLUSTER_2_NAME

Configure endpoint discovery between clusters

Configure endpoint discovery for cross-cluster load balancing by using the following commands. This step performs these tasks:

  • The istioctl command creates a secret that grants access to the Kube API Server for a cluster.
  • The kubectl command applies the secret to another cluster, so that the second cluster can read service endpoints from the first.
istioctl x create-remote-secret --context=${CTX_1} --name=${CLUSTER_1_NAME} | \
  kubectl apply -f - --context=${CTX_2}
istioctl x create-remote-secret --context=${CTX_2} --name=${CLUSTER_2_NAME} | \
  kubectl apply -f - --context=${CTX_1}

Verify your deployment

This section explains how to deploy a sample HelloWorld service to your multi- cluster environment to verify that cross-cluster load balancing works.

Enable sidecar injection

  1. Use the following command to locate the revision label value from the istiod service, which you use in later steps.

    kubectl -n istio-system get pods -l app=istiod --show-labels

    The output looks similar to the following:

    NAME                                READY   STATUS    RESTARTS   AGE   LABELS
    istiod-asm-173-3-5788d57586-bljj4   1/1     Running   0          23h   app=istiod,istio.io/rev=asm-173-3,istio=istiod,pod-template-hash=5788d57586
    istiod-asm-173-3-5788d57586-vsklm   1/1     Running   1          23h   app=istiod,istio.io/rev=asm-173-3,istio=istiod,pod-template-hash=5788d57586
    

    In the output, under the LABELS column, note the value of the istiod revision label, which follows the prefix istio.io/rev=. In this example, the value is asm-173-3. Use the revision value in the steps in the next section.

Install the HelloWorld service

Create the sample namespace and the Service Definition in each cluster. In the following command, substitute REVISION with the istiod revision label that you noted from the previous step.

for CTX in ${CTX_1} ${CTX_2}
  do
    kubectl create --context=${CTX} namespace sample
    kubectl label --context=${CTX} namespace sample \
      istio-injection- istio.io/rev=REVISION --overwrite
    kubectl create --context=${CTX} \
      -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \
      -l app=helloworld -n sample
  done

Deploy HelloWorld v1 and v2 to each cluster

  1. Deploy HelloWorld v1 to CLUSTER_1 and v2 to CLUSTER_2, which helps later to verify cross-cluster load balancing:

    kubectl create --context=${CTX_1} \
      -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \
      -l app=helloworld -l version=v1 -n sample
    kubectl create --context=${CTX_2} \
      -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \
      -l app=helloworld -l version=v2 -n sample
  2. Confirm HelloWorld v1 and v2 are running using the following commands. Verify that the output is similar to that shown.:

    kubectl get pod --context=${CTX_1} -n sample
    NAME                            READY     STATUS    RESTARTS   AGE
    helloworld-v1-86f77cd7bd-cpxhv  2/2       Running   0          40s
    kubectl get pod --context=${CTX_2} -n sample
    NAME                            READY     STATUS    RESTARTS   AGE
    helloworld-v2-758dd55874-6x4t8  2/2       Running   0          40s

Deploy the Sleep service

  1. Deploy the Sleep service to both clusters. This pod generates artificial network traffic for demonstration purposes:

    for CTX in ${CTX_1} ${CTX_2}
      do
        kubectl apply --context=${CTX} \
          -f ${PROJECT_DIR}/samples/sleep/sleep.yaml -n sample
      done
  2. Wait for the Sleep service to start in each cluster. Verify that the output is similar to that shown:

    kubectl get pod --context=${CTX_1} -n sample -l app=sleep
    NAME                             READY   STATUS    RESTARTS   AGE
    sleep-754684654f-n6bzf           2/2     Running   0          5s
    kubectl get pod --context=${CTX_2} -n sample -l app=sleep
    NAME                             READY   STATUS    RESTARTS   AGE
    sleep-754684654f-dzl9j           2/2     Running   0          5s

Verify cross-cluster load balancing

Call the HelloWorld service several times and check the output to verify alternating replies from v1 and v2:

kubectl exec --context=${CTX_1} -it -n sample -c sleep \
   $(kubectl get pod --context=${CTX_1} -n sample -l \
   app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl \
   helloworld.sample:5000/hello
Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8
Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv
...
kubectl exec --context=${CTX_2} -it -n sample -c sleep \
   $(kubectl get pod --context=${CTX_2} -n sample -l \
   app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl \
   helloworld.sample:5000/hello

Verify that the output is similar to that shown:

Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8
Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv
...

Congratulations, you've verified your load-balanced, multi-cluster Anthos Service Mesh!

Clean up HelloWorld service

When you finish verifying load balancing, remove the HelloWorld and Sleep service from your cluster.

kubectl delete ns sample --context ${CTX_1}
kubectl delete ns sample --context ${CTX_2}