Building a multi-cluster service mesh on GKE using multi-primary control-plane multi-network architecture

This tutorial describes how to deploy applications across multiple Kubernetes clusters by using an Istio multi-cluster service mesh. An Istio multi-cluster service mesh lets Services that are running on multiple Kubernetes clusters securely communicate with one another. Kubernetes clusters could be running anywhere, even in different cloud platforms—for example, Google Kubernetes Engine (GKE) clusters running in Google Cloud, or a Kubernetes cluster running in an on-premises data center.

This tutorial is intended for operators who want to create a service mesh across multiple GKE clusters across multiple networks. Base knowledge of Kubernetes is required, including Deployments, Services, Ingress, and so on. Base knowledge of Istio is good to have but not required.

Istio is an open source implementation of a service mesh that lets you discover, dynamically route to, and securely connect to Services running on Kubernetes clusters. Istio also provides a policy-driven framework for routing, load balancing, throttling, telemetry, circuit breaking, authenticating, and authorizing service calls in the mesh with few or no changes in your application code. When Istio is installed in a Kubernetes cluster, it uses the Kubernetes Service registry to automatically discover and create a service mesh of interconnected services (or microservices) running in multiple GKE clusters. Istio uses Envoy sidecar proxies running in each Pod to manage Pod-to-Pod traffic routing and security, and to provide observability for all microservices and workloads that are running in the cluster.

Microservices running in one Kubernetes cluster might need to talk to microservices running in other Kubernetes clusters. For example, microservices might need to talk across regions, and environments or microservice owners might maintain their own Kubernetes clusters. Istio lets you create a service mesh beyond a single Kubernetes cluster to include microservices running in remote clusters and even external microservices running in VMs, outside of Kubernetes.

Istio provides two main configurations for multi-cluster deployments:

In a multi-primary control-plane configuration, each cluster has its own Istio control-plane installation and each control plane manages its own local service mesh. Using Istio gateways, a common root Certificate Authority (CA), and automatic service discovery across multiple GKE clusters, you configure a single logical service mesh that is composed of participating microservices on each cluster. This results in every cluster managing its own multi-cluster service mesh with all cluster inbound access going through the Istio east-west gateway. This approach has no special networking requirements as long as the Istio east-west gateways are reachable across all clusters.

In this tutorial, you deploy Istio in two GKE clusters using the multi-primary control-plane architecture. For this tutorial, you use a demo microservices app called Online Boutique that is split across two GKE clusters. To see the language for each microservice, see the README page.

You build the following architecture inside a Cloud project.

Istio deployed in two GKE clusters using the multi–control-plane architecture.

In this architecture, you have a west cluster and a central cluster in two separate networks (or VPCs), each with an Istio east-west gateway. The clusters communicate with various microservices both locally (in the same cluster) and nonlocally (in the other cluster).

Objectives

  • Create two GKE clusters, west and central, in two different regions and two different VPCs.
  • Install Istio in multi-primary mode on both GKE clusters.
  • Install the Online Boutique app split across both clusters.
  • Inspect the service mesh in both clusters.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. Enable the GKE and Cloud Source Repositories APIs.

    Enable the APIs

Setting up your environment

You run all of the terminal commands in this tutorial from Cloud Shell.

  1. In the Google Cloud Console, open Cloud Shell:

    OPEN Cloud Shell

  2. Download the required files for this tutorial by cloning the following GitHub repository:

    cd $HOME
    git clone https://github.com/GoogleCloudPlatform/istio-multicluster-gke.git
    
  3. Make the repository folder your $WORKDIR folder from which you do all the tasks related to this tutorial:

    cd $HOME/istio-multicluster-gke
    WORKDIR=$(pwd)
    

    You can delete the folder when you're finished with the tutorial.

  4. Install kubectx/kubens:

    git clone https://github.com/ahmetb/kubectx $WORKDIR/kubectx
    export PATH=$PATH:$WORKDIR/kubectx
    

    These tools make it easier to work with multiple Kubernetes clusters by switching contexts or namespaces.

Creating VPCs and GKE clusters

In this section, you create two VPCs, and two GKE clusters, one in each VPC. You need two distinct VPCs in order to demonstrate that no additional networking requirements are needed for the Istio multi-primary implementation on multiple networks. Service-to-Service traffic between clusters flows securely over the internet. Routing traffic over the public internet between clusters provides the following advantages:

  • It supports overlapping IP addresses across clusters. Node, Pod, and Service IP addresses can overlap across clusters.
  • It requires no additional connectivity between clusters, and it doesn't require peering, interconnect, or VPN between cluster networks.
  • It allows clusters to exist in multiple environments. This means that clusters can exist in Google Cloud and in on-premises data centers where operators might not have control over the networking or IP addresses, but where operators might still need to securely connect to microservices running in Kubernetes clusters.

Note that even across clusters, you can help secure Service-to-Service communication between clusters by using mutual TLS (mTLS) connections. By verifying valid certificates issued by Citadel, these connections help avoid man-in-the-middle attacks.

You can use private IP addresses to communicate between clusters. However, this approach requires additional networking design consideration. For example, you could include non-overlapping IP addressing between all clusters participating in the multi-cluster mesh. You could also ensure that all Pods can communicate over the private (RFC 1918) address space, which means proper firewall rules within a Google Cloud global VPC, or within an interconnect or VPN if you are connecting to non-Google Cloud networks. For this tutorial, you configure the architecture that focuses on service-to-service secure communication over the public internet, giving you greater networking flexibility.

  1. In Cloud Shell, create the VPCs:

    gcloud compute networks create vpc-west --subnet-mode=auto
    gcloud compute networks create vpc-central --subnet-mode=auto
    
  2. Set the KUBECONFIG variable to use a new kubeconfig file for this tutorial:

    export KUBECONFIG=${WORKDIR}/istio-kubeconfig
    
  3. Create two GKE clusters, one named west in the us-west2-a zone in the vpc-west network, and one named central in the us-central1-a zone in the vpc-central network:

    gcloud container clusters create west --zone us-west2-a \
        --machine-type "e2-standard-2" --disk-size "100" \
        --scopes "https://www.googleapis.com/auth/compute",\
    "https://www.googleapis.com/auth/devstorage.read_only",\
    "https://www.googleapis.com/auth/logging.write",\
    "https://www.googleapis.com/auth/monitoring",\
    "https://www.googleapis.com/auth/servicecontrol",\
    "https://www.googleapis.com/auth/service.management.readonly",\
    "https://www.googleapis.com/auth/trace.append" \
        --num-nodes "4" --network "vpc-west" --async
    
    gcloud container clusters create central --zone us-central1-a \
        --machine-type "e2-standard-2" --disk-size "100" \
        --scopes "https://www.googleapis.com/auth/compute",\
    "https://www.googleapis.com/auth/devstorage.read_only",\
    "https://www.googleapis.com/auth/logging.write",\
    "https://www.googleapis.com/auth/monitoring",\
    "https://www.googleapis.com/auth/servicecontrol",\
    "https://www.googleapis.com/auth/service.management.readonly",\
    "https://www.googleapis.com/auth/trace.append" \
        --num-nodes "4" --network "vpc-central"
    
  4. Wait a few minutes until both clusters are created. Verify that the status for each cluster is RUNNING:

    gcloud container clusters list
    

    The output is similar to the following:

    NAME     LOCATION       MASTER_VERSION    MASTER_IP        MACHINE_TYPE   NODE_VERSION      NUM_NODES  STATUS
    central  us-central1-a  1.17.14-gke.1600  104.197.127.140  e2-standard-2  1.17.14-gke.1600  4          RUNNING
    west     us-west2-a     1.17.14-gke.1600  34.94.217.4      e2-standard-2  1.17.14-gke.1600  4          RUNNING
    
  5. Connect to both clusters to generate entries in the kubeconfig file:

    export PROJECT_ID=$(gcloud info --format='value(config.project)')
    gcloud container clusters get-credentials west --zone us-west2-a --project ${PROJECT_ID}
    gcloud container clusters get-credentials central --zone us-central1-a --project ${PROJECT_ID}
    

    You use the kubeconfig file to create authentication to clusters by creating a user and context for each cluster. After you create the kubeconfig file, you can quickly switch context between clusters.

  6. Use kubectx to rename the context names for convenience:

    kubectx west=gke_${PROJECT_ID}_us-west2-a_west
    kubectx central=gke_${PROJECT_ID}_us-central1-a_central
    
  7. Give yourself (your Google user) the cluster-admin role for both clusters:

    kubectl create clusterrolebinding user-admin-binding \
        --clusterrole=cluster-admin --user=$(gcloud config get-value account) \
        --context west
    kubectl create clusterrolebinding user-admin-binding \
        --clusterrole=cluster-admin --user=$(gcloud config get-value account) \
        --context central
    

    This role lets you perform administrative tasks on these clusters.

Configuring certificates on both clusters

In Istio, Citadel is Istio's certificate authority (CA) and is responsible for signing and distributing certificates to all Envoy proxies (workload sidecar proxies, and ingress, east-west and egress gateway proxies) in the service mesh. By default, Citadel generates a self-signed root certificate and key, and uses them to sign the workload certificates. Citadel can also use an operator-specified certificate and key to sign workload certificates. In this tutorial, the west and central clusters maintain separate service meshes with separate Citadel services signing their respective workloads.

If you want to establish trust between microservices across clusters, both Citadel signing (CA) certificates must be signed by a common root certificate authority (root CA). Workloads also need a certificate chain file to verify the chain of trust of all intermediate CAs between the Citadel signing certificate and the root CA. This configuration is done by using a secret in the Kubernetes cluster. The certificate files are provided to you as part of this tutorial. In your production environment, you can use certificates generated by your PKI systems or security team (for example, you can act as your own CA). The secret created is used by Citadel and has the following properties:

  • The secret must be named cacert.
  • The secret is created from four certificate files (provided by the Istio code for this tutorial), which must be named as follows:
    • root-cert.pem contains the root CA. This file is the same for Citadel services in both west and central clusters. Both Citadel certificates are signed by this root CA.
    • ca-cert.pem and ca-key.pem are the signing (CA) certificate and private key for the Citadel service. Both Citadel certificates must be signed by the root CA (root-cert.pem). ca-cert.pem is used to sign the workloads within each cluster.
    • cert-chain.pem is the chain of trust between the workload certificates and the root CA. In this example, cert-chain.pem contains only the ca-cert.pem certificate; hence, these files are identical. This file establishes trust between microservices that are running across clusters.

The default Citadel installation sets command-line options to configure the location of certificates and keys based on the predefined secret and file names used in the command—that is, a secret named cacert, a root certificate in a file named root-cert.pem, a Citadel key in ca-key.pem, and so on.

  1. In Cloud Shell, download Istio:

    cd ${WORKDIR}
    export ISTIO_VERSION=1.9.0
    curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} TARGET_ARCH=x86_64 sh -
    cd istio-${ISTIO_VERSION}
    export PATH=$PWD/bin:$PATH
    
  2. In Cloud Shell, create the secret by using the appropriate certificate files:

    for cluster in $(kubectx)
    do
      kubectl --context $cluster create namespace istio-system
      kubectl --context $cluster create secret generic cacerts -n istio-system \
        --from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/ca-cert.pem \
        --from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/ca-key.pem \
        --from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/root-cert.pem \
        --from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/cert-chain.pem;
      done
    

Installing Istio

In this section you install Istio on the west and central clusters.

  1. In Cloud Shell, set the default network for the west cluster.

    kubectl --context=west label namespace istio-system topology.istio.io/network=network1
    
  2. Create the Istio configuration for the west cluster with a dedicated east-west gateway:

    cd ${WORKDIR}
    cat <<EOF > istio-west.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      values:
        global:
          meshID: mesh1
          multiCluster:
            clusterName: west
          network: network1
      components:
        ingressGateways:
          - name: istio-eastwestgateway
            label:
              istio: eastwestgateway
              app: istio-eastwestgateway
              topology.istio.io/network: network1
            enabled: true
            k8s:
              env:
                # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
                - name: ISTIO_META_ROUTER_MODE
                  value: "sni-dnat"
                # traffic through this gateway should be routed inside the network
                - name: ISTIO_META_REQUESTED_NETWORK_VIEW
                  value: network1
              service:
                ports:
                  - name: status-port
                    port: 15021
                    targetPort: 15021
                  - name: tls
                    port: 15443
                    targetPort: 15443
                  - name: tls-istiod
                    port: 15012
                    targetPort: 15012
                  - name: tls-webhook
                    port: 15017
                    targetPort: 15017
    EOF
    
  3. Apply the configuration to the west cluster:

    istioctl install --context=west -f istio-west.yaml
    

    Press y to continue.

  4. Inspect the deployments in the istio-system namespace.

    kubectl --context=west -n istio-system get deployments
    

    The output looks like the following:

    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    istio-eastwestgateway   1/1     1            1           2m11s
    istio-ingressgateway    1/1     1            1           8m43s
    istiod                  1/1     1            1           8m56s
    
  5. Wait for the east-west gateway to be assigned an external IP address:

    kubectl --context=west get svc istio-eastwestgateway -n istio-system
    

    The output looks like the following:

    NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                                           AGE
    istio-eastwestgateway   LoadBalancer   10.3.241.43   34.94.214.249   15021:30369/TCP,15443:30988/TCP,15012:31358/TCP,15017:32625/TCP   3m42s
    
  6. Since the clusters are on separate networks, you need to expose all services (*.local) on the east-west gateway in both clusters. Since the east-west gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

    kubectl --context=west apply -n istio-system -f \
    ${WORKDIR}/istio-${ISTIO_VERSION}/samples/multicluster/expose-services.yaml
    
  7. Set the default network for the central cluster.

    kubectl --context=central label namespace istio-system topology.istio.io/network=network2
    
  8. Create the Istio configuration for the central cluster with a dedicated east-west gateway:

    cd ${WORKDIR}
    cat <<EOF > istio-central.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      values:
        global:
          meshID: mesh1
          multiCluster:
            clusterName: central
          network: network2
      components:
        ingressGateways:
          - name: istio-eastwestgateway
            label:
              istio: eastwestgateway
              app: istio-eastwestgateway
              topology.istio.io/network: network2
            enabled: true
            k8s:
              env:
                # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
                - name: ISTIO_META_ROUTER_MODE
                  value: "sni-dnat"
                # traffic through this gateway should be routed inside the network
                - name: ISTIO_META_REQUESTED_NETWORK_VIEW
                  value: network2
              service:
                ports:
                  - name: status-port
                    port: 15021
                    targetPort: 15021
                  - name: tls
                    port: 15443
                    targetPort: 15443
                  - name: tls-istiod
                    port: 15012
                    targetPort: 15012
                  - name: tls-webhook
                    port: 15017
                    targetPort: 15017
    EOF
    
  9. Apply the configuration to the central cluster:

    istioctl install --context=central -f istio-central.yaml
    

    Press y to continue.

  10. Inspect the deployments in the istio-system namespace.

    kubectl --context=central -n istio-system get deployments
    

    The output looks like the following:

    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    istio-eastwestgateway   1/1     1            1           2m11s
    istio-ingressgateway    1/1     1            1           8m43s
    istiod                  1/1     1            1           8m56s
    
  11. Wait for the east-west gateway to be assigned an external IP address:

    kubectl --context=central get svc istio-eastwestgateway -n istio-system
    

    The output looks like the following:

    NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                                                           AGE
    istio-eastwestgateway   LoadBalancer   10.3.250.201   35.232.125.62   15021:30810/TCP,15443:31125/TCP,15012:30519/TCP,15017:30743/TCP   37s
    
  12. Expose all services (*.local) on the east-west gateway in the central cluster.

    kubectl --context=central apply -n istio-system -f \
    ${WORKDIR}/istio-${ISTIO_VERSION}/samples/multicluster/expose-services.yaml
    

Enabling endpoint discovery

  1. Install a remote secret in west cluster that provides access to central cluster's API server.

    istioctl x create-remote-secret \
    --context=central \
    --name=central | \
    kubectl apply -f - --context=west
    
  2. Install a remote secret in central cluster that provides access to west cluster's API server.

    istioctl x create-remote-secret \
    --context=west \
    --name=west | \
    kubectl apply -f - --context=central
    

Deploying the Online Boutique app

In this section, you install the Online Boutique app to both clusters. Online Boutique consists of 10 microservices written in different programming languages. You split these microservices across the central and west clusters, as shown in the following architecture.

Microservices are split across the `central` and `west` clusters, with overlapped service meshes. In the diagram, service A

  1. Create namespaces for the Online Boutique app in both clusters:

    kubectl --context central create namespace online-boutique
    kubectl --context west create namespace online-boutique
    
  2. Label the namespaces for automatic Istio sidecar proxy injection:

    kubectl --context central label namespace online-boutique istio-injection=enabled
    kubectl --context west label namespace online-boutique istio-injection=enabled
    
  3. Deploy the Online Boutique app on the west and central clusters:

    kubectl --context central -n online-boutique apply -f $WORKDIR/istio-multi-primary/central
    kubectl --context west -n online-boutique apply -f $WORKDIR/istio-multi-primary/west
    

    The Online Boutique app takes a few minutes for all Deployments to be up and running.

  4. Wait until all deployments are available on both clusters.

    # In central cluster
    kubectl --context=central -n online-boutique wait --for=condition=available deployment emailservice --timeout=5m
    kubectl --context=central -n online-boutique wait --for=condition=available deployment checkoutservice --timeout=5m
    kubectl --context=central -n online-boutique wait --for=condition=available deployment shippingservice --timeout=5m
    kubectl --context=central -n online-boutique wait --for=condition=available deployment paymentservice --timeout=5m
    kubectl --context=central -n online-boutique wait --for=condition=available deployment adservice --timeout=5m
    kubectl --context=central -n online-boutique wait --for=condition=available deployment currencyservice --timeout=5m
    
    # In west cluster
    kubectl --context=west -n online-boutique wait --for=condition=available deployment frontend --timeout=5m
    kubectl --context=west -n online-boutique wait --for=condition=available deployment recommendationservice --timeout=5m
    kubectl --context=west -n online-boutique wait --for=condition=available deployment productcatalogservice --timeout=5m
    kubectl --context=west -n online-boutique wait --for=condition=available deployment cartservice --timeout=5m
    kubectl --context=west -n online-boutique wait --for=condition=available deployment redis-cart --timeout=5m
    kubectl --context=west -n online-boutique wait --for=condition=available deployment loadgenerator --timeout=5m
    

    The output for all deployments looks like the following:

    deployment.apps/frontend condition met
    

Accessing the Online Boutique app

You can access Online Boutique app from the istio-ingressgatway public IP address from either cluster.

  1. Get the public IP address of the istio-ingressgateway service from both clusters.

    kubectl --context west get -n istio-system service istio-ingressgateway -o json | jq -r '.status.loadBalancer.ingress[0].ip'
    kubectl --context central get -n istio-system service istio-ingressgateway -o json | jq -r '.status.loadBalancer.ingress[0].ip'
    

    The output shows the external IP address.

  2. Copy and paste the Istio ingress gateway IP address from either cluster into a web browser tab, and then refresh the page. You see the Online Boutique home page.

    Hipster app home page showing pictures of various products such as bikes, cameras, typewriters, and so on.

    Navigate around the app, browse various products, place them in your cart, check out, and so on.

    You see that the Online Boutique is fully functional and running across two Kubernetes clusters in two environments.

Monitoring the service mesh

You can use Kiali to monitor and visualize the service mesh. Kiali is a service-mesh observability tool that's installed as part of the Istio installation.

  1. Deploy Prometheus and Kiali to the west cluster.

    kubectl --context west apply -f https://raw.githubusercontent.com/istio/istio/release-${ISTIO_VERSION:0:3}/samples/addons/prometheus.yaml
    kubectl --context west apply -f https://raw.githubusercontent.com/istio/istio/release-${ISTIO_VERSION:0:3}/samples/addons/kiali.yaml
    

    If the Kiali install fails with errors, try running the same command again. There might be some timing issues which will be resolved when the command is run again.

  2. Ensure that the Prometheus and Kiali Pods are running:

    kubectl --context west get pods -n istio-system
    

    The output looks like the following:

    kiali-6d5c4bbb64-wpsqx                   1/1     Running   0          72s
    prometheus-6d87d85c88-h2cwl              2/2     Running   0          92s
    
  3. In Cloud Shell, expose Kiali on the west cluster:

    kubectl --context west port-forward svc/kiali 8080:20001 -n istio-system >> /dev/null &
    
  4. Open the Kiali web interface on the west cluster. In Cloud Shell, select Web Preview, and then select Preview on port 8080.

  5. From the left menu, select Graph.

  6. From the Select a namespace drop-down list, select online-boutique.

  7. From the menu under Graph, select Service graph.

  8. Optionally, from the Display menu, select Traffic Animation to see loadgenerator generating traffic to your app.

    Screenshot that shows an Internal Service Entry with the symbol of a lock next to it.

You have successfully installed an application split across multiple clusters. The microservices are securely communicating across the clusters using the Istio east-west gateway with mTLS. Each cluster maintains and controls its own Istio controlplane, avoiding any single point of failure.

Clean up

After you've finished the tutorial, clean up the resources you created on Google Cloud so you won't be billed for them in the future. The following sections describe how to delete these resources.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Reset kubeconfig

  1. Reset your kubeconfig file:

    unset KUBECONFIG
    rm istio-kubeconfig
    

What's next