Version 1.11

Set up a multi-cluster mesh outside Google Cloud

This guide explains how to set up a multi-cluster mesh for the following platforms:

  • Anthos clusters on VMware
  • Anthos on bare metal
  • Anthos clusters on AWS
  • Amazon EKS

This guide shows how to set up two clusters, but you can extend this process to incorporate any number of clusters into your mesh.

Before you begin

This guide assumes you installed Anthos Service Mesh using asmcli install. You need asmcli and the configuration package that asmcli downloads to the directory that you specified in --output_dir when you ran asmcli install. If need to get set up, follow the steps in Get started to:

You need access to the kubeconfig files for all the clusters that you are setting up in the mesh.

Set up environment variables and placeholders

You need the following environment variables when you install the east-west gateway.

  1. Create an environment variable for the project number. In the following command, replace FLEET_PROJECT_ID with the the project ID of the fleet host project.

    export PROJECT_NUMBER=$(gcloud projects describe FLEET_PROJECT_ID --format="value(projectNumber)")
    
  2. Create an environment variable for the mesh identifier.

    export MESH_ID="proj-${PROJECT_NUMBER}"
    
  3. Create environment variables for the cluster names in the format that asmcli requires:

    export CLUSTER_1="cn-FLEET_PROJECT_ID-global-CLUSTER_NAME_1"
    export CLUSTER_2="cn-FLEET_PROJECT_ID-global-CLUSTER_NAME_2"
    

Install the east-west gateway

In the following commands:

  • Replace CLUSTER_NAME_1 and CLUSTER_NAME_2 with the names of your clusters.

  • Replace PATH_TO_KUBECONFIG_1 and PATH_TO_KUBECONFIG_2 with the kubeconfig files for your clusters.

Mesh CA

  1. Install a gateway in cluster1 that is dedicated to east-west traffic to $CLUSTER_2. By default, this gateway will be public on the Internet. Production systems might require additional access restrictions, for example firewall rules, to prevent external attacks.

    asm/istio/expansion/gen-eastwest-gateway.sh \
        --mesh ${MESH_ID} \
        --cluster ${CLUSTER_1} \
        --network default \
        --revision asm-1112-17 | \
        istioctl --kubeconfig=PATH_TO_KUBECONFIG_1 install -y --set spec.values.global.pilotCertProvider=kubernetes -f -
    
  2. Install a gateway in $CLUSTER_2 that is dedicated to east-west traffic for $CLUSTER_1.

    asm/istio/expansion/gen-eastwest-gateway.sh \
        --mesh ${MESH_ID} \
        --cluster ${CLUSTER_2} \
        --network default \
        --revision asm-1112-17 | \
        istioctl install --kubeconfig=PATH_TO_KUBECONFIG_2 install -y --set spec.values.global.pilotCertProvider=kubernetes -f -
    

Istio CA

  1. Install a gateway in cluster1 that is dedicated to east-west traffic to $CLUSTER_2. By default, this gateway will be public on the Internet. Production systems might require additional access restrictions, for example firewall rules, to prevent external attacks.

    asm/istio/expansion/gen-eastwest-gateway.sh \
        --mesh ${MESH_ID} \
        --cluster ${CLUSTER_1} \
        --network default \
        --revision asm-1112-17 | \
        istioctl --kubeconfig=PATH_TO_KUBECONFIG_1 install -y -f -
    
  2. Install a gateway in $CLUSTER_2 that is dedicated to east-west traffic for $CLUSTER_1.

    asm/istio/expansion/gen-eastwest-gateway.sh \
        --mesh ${MESH_ID} \
        --cluster ${CLUSTER_2} \
        --network default \
        --revision asm-1112-17 | \
        istioctl --kubeconfig=PATH_TO_KUBECONFIG_2 install -y -f -
    

Exposing services

Since the clusters are on separate networks, you need to expose all services (*.local) on the east-west gateway in both clusters. While this gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

  1. Expose services via the east-west gateway for CLUSTER_NAME_1.

    kubectl --kubeconfig=PATH_TO_KUBECONFIG_1 apply -n istio-system -f \
        asm/istio/expansion/expose-services.yaml
    
  2. Expose services via the east-west gateway for CLUSTER_NAME_2.

    kubectl --kubeconfig=PATH_TO_KUBECONFIG_2 apply -n istio-system -f \
        asm/istio/expansion/expose-services.yaml
    

Enable endpoint discovery

Run the asmcli create-mesh command to enable endpoint discovery. This example only shows two clusters, but you can run the command to enable endpoint discovery on additional clusters, subject to the GKE Hub service limit.

  ./asmcli create-mesh \
      FLEET_PROJECT_ID \
      PATH_TO_KUBECONFIG_1 \
      PATH_TO_KUBECONFIG_2