Setting up a multi-cluster Ingress


If you have an application running on multiple Google Kubernetes Engine clusters located in different regions, you can route traffic to a cluster in the region closest to the user.

This page uses the kubemci command-line tool to configure a Cloud HTTP Load Balancer using a Kubernetes Ingress to route traffic across multiple clusters in different regions.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have installed the Cloud SDK.
  • Initialize gcloud with the following command:

    gcloud init
    • Choose the account you would like to use or login with a new account.
    • Enter your project ID or create a new project.
    • Configure a default Compute Region and Zone.
  • Authenticate and allow Google Cloud SDK to view and manage your data across Google Cloud Platform services:

    gcloud auth login
  • Authenticate and allow Google Auth Library to view and manage your data across Google Cloud Platform services:

    gcloud auth application-default login
  • Install the kubectl command-line tool:

    gcloud components install kubectl
  • Ensure that you have two or more GKE clusters running in the same GCP project and network.

Download the kubemci command-line tool

kubemci is the primary command-line tool used to create and configure a multi-cluster ingress.

Download one of the binaries and place it in an executable path:

Ensure that the binary is executable. This can be done in the directory of the binary with the following command:

chmod +x ./kubemci

Creating a multi-cluster ingress

Prepare your clusters

For each Service you are planning to use in the multi-cluster ingress, it must be configured the same across all of the clusters. Specifically, that Service must:

  • Have the same name in all of the clusters.
  • Be in the same namespace in all of the clusters.
  • Be of type NodePort.
  • Use the same port number for all of the clusters.

    For example, if your Ingress requires two services, service-foo and service-bar, then:

    • service-foo is a nodePort on port 30016 in all of the clusters.
    • service-bar is a nodePort on port 30017 in all of the clusters.

Refer to the Deploy a nodePort service to each cluster section for more detailed instructions.

Create a kubeconfig file

The kubemci command-line tool requires a kubeconfig file that contains the cluster connection information of the GKE clusters that are going to receive traffic.

Run the following command for each cluster to create a kubeconfig file containing credentials for all of the clusters:

gcloud container clusters get-credentials \

For example:

KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=us-east4-a  cluster-us-east
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=europe-west1-c  cluster-eu-west
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=asia-east1-b  cluster-apac-east

Deploy a NodePort service to each cluster

Place the Service and application deployment manifest files in the same directory for easy deployment. For example, the following files would both be placed in the app/ directory:

Example Deployment manifest file

apiVersion: apps/v1
kind: Deployment
  name: my-deployment
      app: my-app
  replicas: 2
        app: my-app
      - name: hello-app

Example Service manifest file

apiVersion: v1
kind: Service
    app: my-app
  name: service-foo
  - port: 80
    protocol: TCP
    targetPort: 8080
    name: http
    nodePort: 30061
    app: my-app
  type: NodePort

You can iterate through all of the clusters and get the contexts to deploy the application, along with its nodePort service, by running the following loop:

for ctx in $(kubectl config get-contexts -o name); do
  kubectl --context="${ctx}" create -f [PATH/TO/APP/FILES]

Reserve a static IP

Request a static IP for the multi-cluster Ingress on Google Cloud Platform by running the following command:

gcloud compute addresses create --global [IP_RESOURCE_NAME]

For example, if you name the IP resource my-mci-ip, it would be:

gcloud compute addresses create --global my-mci-ip

Prepare the Ingress resource

In each cluster, the spec of the Ingress resource must have the metadata annotations field and the backend field configured with values for the multi-cluster Ingress.

The following is an example Ingress resource file, mcingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
  name: ingress-foo
  annotations: my-mci-ip gce-multi-cluster
    serviceName: service-foo
    servicePort: 80

Add or modify the annotations:

  • Set the value of to your [IP_RESOURCE_NAME].
  • Set the value of to gce-multi-cluster.

Set a backend service:

  • Set the value of serviceName to the name of the default service.
  • Set the value of servicePort to the port of the default service.

Deploy the multi-cluster Ingress

Use kubemci to create the multi-cluster Ingress with the following command:

kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[optional PROJECT_ID] \
For example:
kubemci create my-mci \
    --ingress=mcingress.yaml \
    --gcp-project=my-gcp-project \

It outputs a series of messages to confirm the creation process, for example:

Ensuring health checks
Health check mci1-hc-30061--my-mci created successfully
Backend service mci1-be-30061--my-mci created successfully
URL Map mci1-um--my-mci created successfully
Target http proxy mci1-tp--my-mci created successfully
Forwarding rule mci1-fw--my-mci created successfully
Firewall rule mci1-fr--my-mci created successfully


Getting the status of a multi-cluster Ingress

You can list all the existing multi-cluster Ingresses that were created using kubemci by running the following command:

kubemci list --gcp-project=[PROJECT_ID]

It outputs a list of multi-cluster Ingresses similar the following:

NAME           IP              CLUSTERS
my-mci   gke_my-gcp-project_asia-east1-b_cluster-asia-east, gke_my-gcp-project_europe-west1-c_cluster-eu-west, gke_my-gcp-project_us-east4-a_cluster-us-east

You can then get the status of a specific multi-cluster Ingress with the following command:

kubemci get-status [INGRESS_NAME] --gcp-project=[PROJECT_ID]

It outputs a status message similar to the following:

Load balancer my-mci has IPAddress and is spread across 3 clusters
(gke_project-name_asia-east1-b_cluster-asia-east, gke_project-name_europe-west1-c_cluster-eu-west,

Updating an existing multi-cluster Ingress

using --force flag

To update an existing multi-cluster Ingress, run the kubemci create command with the --force flag:

kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE] \

delete, recreate

Alternatively, to update a multi-cluster Ingress, you can delete it and then create it again:

kubemci delete [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \

Removing clusters from an existing multi-cluster Ingress

To remove clusters from an existing multi-cluster Ingress, first create a kubeconfig that contains a list of clusters that are to be removed from the cluster, then run the kubemci remove-clusters command:

kubemci remove-clusters [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \

Deleting a multi-cluster Ingress

To delete a multi-cluster Ingress, use the kubemci delete command:

kubemci delete [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \

It outputs a series of messages to confirm the deletion process, for example:

Deleting Ingress from cluster: gke_project-name_asia-east1-b_cluster-asia-east...
firewall rule mci1-fr--my-mci deleted successfully
http forwarding rule mci1-fw--my-mci deleted successfully
target HTTPS proxy mci1-tps--my-mci deleted successfully
ssl cert mci1-ssl--my-mci deleted successfully
url map mci1-um--my-mci deleted successfully
Successfully deleted all backend services
Successfully deleted all health checks

What's next