This guide explains how to join two clusters into a single Anthos Service Mesh using Mesh CA or Citadel, and enable cross-cluster load balancing. You can easily extend this process to incorporate any number of clusters into your mesh.
A multi-cluster Anthos Service Mesh configuration can solve several crucial enterprise scenarios, such as scale, location, and isolation. For more information, see Multi-cluster use cases. In addition, you should optimize your applications to get the most benefit from a service mesh. For more information, see Preparing an application for Anthos Service Mesh.
Prerequisites
This guide assumes that you have two or more Google Cloud GKE clusters that meet the following requirements:
- Anthos Service Mesh version 1.6.8 or higher installed on the clusters.
- If your clusters are in the same project, see Installation overview to install or upgrade your clusters to the required version.
- If your clusters are in different projects, see Multi-project installation and migration to install or upgrade your clusters to the required version.
- If you join clusters that are not in the same project, they must be installed
using the
asm-gcp-multiproject
profile and the clusters must be in a shared VPC configuration together on the same network. In addition, we recommend that you have one project to host the shared VPC, and two service projects for creating clusters. For more information, see Setting up clusters with Shared VPC. - If you use Citadel CA, use the same custom root CA for both clusters.
- If your Anthos Service Mesh is built on private clusters, we recommend
creating a single subnet
in the same VPC, otherwise, you must ensure that:
- The control planes can reach the remote private cluster control planes via the cluster private IPs.
- You can add the calling control planes' IP ranges to the remote private clusters' authorized networks. For more information, see Configure endpoint discovery between private clusters.
Setting project and cluster variables
Set a working folder for convenience. This is the folder in which you downloaded and extracted the Anthos Service Mesh files in the prerequisite step, Preparing to install Anthos Service Mesh.
export PROJECT_DIR=YOUR_WORKING_FOLDER
Create a context variable for each cluster. The context is a string constructed from the cluster project IDs, cluster names, and locations. For the location values, use the cluster's location, for example
us-west2-a
. In this example, a mesh contains one cluster already and you're adding another cluster to the mesh:export CTX_1=gke_CLUSTER_1_PROJECT_ID_CLUSTER_1_LOCATION_CLUSTER_1_NAME export CTX_2=gke_CLUSTER_2_PROJECT_ID_CLUSTER_2_LOCATION_CLUSTER_2_NAME
Configure endpoint discovery between clusters
Configure endpoint discovery for cross-cluster load balancing by using the following commands. This step performs these tasks:
- The
istioctl
command creates a secret that grants access to the Kube API Server for a cluster. - The
kubectl
command applies the secret to another cluster, so that the second cluster can read service endpoints from the first.
istioctl x create-remote-secret --context=${CTX_1} --name=${CLUSTER_1_NAME} | \ kubectl apply -f - --context=${CTX_2}
istioctl x create-remote-secret --context=${CTX_2} --name=${CLUSTER_2_NAME} | \ kubectl apply -f - --context=${CTX_1}
Verify your deployment
This section explains how to deploy a sample HelloWorld
service to your multi-
cluster environment to verify that cross-cluster load balancing works.
Enable sidecar injection
Use the following command to locate the revision label value from the
istiod
service, which you use in later steps.kubectl -n istio-system get pods -l app=istiod --show-labels
The output looks similar to the following:
NAME READY STATUS RESTARTS AGE LABELS istiod-asm-173-3-5788d57586-bljj4 1/1 Running 0 23h app=istiod,istio.io/rev=asm-173-3,istio=istiod,pod-template-hash=5788d57586 istiod-asm-173-3-5788d57586-vsklm 1/1 Running 1 23h app=istiod,istio.io/rev=asm-173-3,istio=istiod,pod-template-hash=5788d57586
In the output, under the
LABELS
column, note the value of theistiod
revision label, which follows the prefixistio.io/rev=
. In this example, the value isasm-173-3
. Use the revision value in the steps in the next section.
Install the HelloWorld service
Create the sample namespace and the Service Definition in each cluster.
Create the sample namespace in each cluster:
kubectl create --context=${CTX_1} namespace sample
kubectl create --context=${CTX_2} namespace sample
Overwrite the revision label:
kubectl label --context=${CTX_1} namespace sample \ istio-injection- istio.io/rev=REVISION --overwrite
kubectl label --context=${CTX_2} namespace sample \ istio-injection- istio.io/rev=REVISION --overwrite
where REVISION is the
istiod
revision label that you previously noted.The output is:
label "istio-injection" not found. namespace/sample labeled
You can safely ignore
label "istio-injection" not found.
Create the HelloWorld service in both clusters:
kubectl create --context=${CTX_1} \ -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \ -l service=helloworld -n sample
kubectl create --context=${CTX_2} \ -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \ -l service=helloworld -n sample
Deploy HelloWorld v1 and v2 to each cluster
Deploy
HelloWorld v1
toCLUSTER_1
andv2
toCLUSTER_2
, which helps later to verify cross-cluster load balancing:kubectl create --context=${CTX_1} \ -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \ -l version=v1 -n sample
kubectl create --context=${CTX_2} \ -f ${PROJECT_DIR}/samples/helloworld/helloworld.yaml \ -l version=v2 -n sample
Confirm
HelloWorld v1
andv2
are running using the following commands. Verify that the output is similar to that shown.:kubectl get pod --context=${CTX_1} -n sample
NAME READY STATUS RESTARTS AGE helloworld-v1-86f77cd7bd-cpxhv 2/2 Running 0 40s
kubectl get pod --context=${CTX_2} -n sample
NAME READY STATUS RESTARTS AGE helloworld-v2-758dd55874-6x4t8 2/2 Running 0 40s
Deploy the Sleep service
Deploy the
Sleep
service to both clusters. This pod generates artificial network traffic for demonstration purposes:for CTX in ${CTX_1} ${CTX_2} do kubectl apply --context=${CTX} \ -f ${PROJECT_DIR}/samples/sleep/sleep.yaml -n sample done
Wait for the
Sleep
service to start in each cluster. Verify that the output is similar to that shown:kubectl get pod --context=${CTX_1} -n sample -l app=sleep
NAME READY STATUS RESTARTS AGE sleep-754684654f-n6bzf 2/2 Running 0 5s
kubectl get pod --context=${CTX_2} -n sample -l app=sleep
NAME READY STATUS RESTARTS AGE sleep-754684654f-dzl9j 2/2 Running 0 5s
Verify cross-cluster load balancing
Call the HelloWorld
service several times and check the output to verify
alternating replies from v1 and v2:
Call the
HelloWorld
service:kubectl exec --context="${CTX_1}" -n sample -c sleep \ "$(kubectl get pod --context="${CTX_1}" -n sample -l \ app=sleep -o jsonpath='{.items[0].metadata.name}')" \ -- curl -sS helloworld.sample:5000/hello
The output is similar to that shown:
Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8 Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv ...
Call the
HelloWorld
service again:kubectl exec --context="${CTX_2}" -n sample -c sleep \ "$(kubectl get pod --context="${CTX_2}" -n sample -l \ app=sleep -o jsonpath='{.items[0].metadata.name}')" \ -- curl -sS helloworld.sample:5000/hello
The output is similar to that shown:
Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8 Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv ...
Congratulations, you've verified your load-balanced, multi-cluster Anthos Service Mesh!
Clean up HelloWorld service
When you finish verifying load balancing, remove the HelloWorld
and Sleep
service from your cluster.
kubectl delete ns sample --context ${CTX_1} kubectl delete ns sample --context ${CTX_2}