Overview
If you have an application running on multiple Google Kubernetes Engine clusters located in different regions, you can route traffic to a cluster in the region closest to the user.
This page uses the kubemci
command-line tool to configure a Cloud HTTP Load Balancer
using a Kubernetes Ingress to route traffic across multiple clusters in
different regions.
Before you begin
To prepare for this task, perform the following steps:
- Ensure that you have installed the Cloud SDK.
Initialize
gcloud
with the following command:gcloud init
- Choose the account you would like to use or login with a new account.
- Enter your project ID or create a new project.
- Configure a default Compute Region and Zone.
Authenticate and allow Google Cloud SDK to view and manage your data across Google Cloud Platform services:
gcloud auth login
Authenticate and allow Google Auth Library to view and manage your data across Google Cloud Platform services:
gcloud auth application-default login
Install the kubectl command-line tool:
gcloud components install kubectl
Ensure that you have two or more GKE clusters running in the same GCP project and network.
Download the kubemci
command-line tool
kubemci
is
the primary command-line tool used to create and configure a multi-cluster ingress.
Download one of the binaries and place it in an executable path:
Ensure that the binary is executable. This can be done in the directory of the binary with the following command:
chmod +x ./kubemci
Creating a multi-cluster ingress
Prepare your clusters
For each Service you are planning to use in the multi-cluster ingress, it must be configured the same across all of the clusters. Specifically, that Service must:
- Have the same name in all of the clusters.
- Be in the same namespace in all of the clusters.
- Be of type
NodePort
. Use the same port number for all of the clusters.
For example, if your Ingress requires two services,
service-foo
andservice-bar
, then:service-foo
is a nodePort on port 30016 in all of the clusters.service-bar
is a nodePort on port 30017 in all of the clusters.
Refer to the Deploy a nodePort service to each cluster section for more detailed instructions.
Create a kubeconfig file
The kubemci
command-line tool requires a kubeconfig file that contains the
cluster connection information of the GKE clusters that are going
to receive traffic.
Run the following command for each cluster to create a kubeconfig file containing credentials for all of the clusters:
KUBECONFIG=[PATH/TO/MCI_KUBECONFIG_FILE] gcloud container clusters get-credentials \ --zone=[ZONE_NAME] [CLUSTER_NAME]
For example:
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=us-east4-a cluster-us-east
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=europe-west1-c cluster-eu-west
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=asia-east1-b cluster-apac-east
Deploy a NodePort service to each cluster
Place the Service and application deployment manifest files in the same directory
for easy deployment. For example, the following files would both be placed in the
app/
directory:
Example Deployment manifest file
apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: selector: matchLabels: app: my-app replicas: 2 template: metadata: labels: app: my-app spec: containers: - name: hello-app image: gcr.io/google-samples/hello-app:1.0
Example Service manifest file
apiVersion: v1 kind: Service metadata: labels: app: my-app name: service-foo spec: ports: - port: 80 protocol: TCP targetPort: 8080 name: http nodePort: 30061 selector: app: my-app type: NodePort
You can iterate through all of the clusters and get the contexts to deploy the application, along with its nodePort service, by running the following loop:
KUBECONFIG=[PATH/TO/MCI_KUBECONFIG_FILE] for ctx in $(kubectl config get-contexts -o name); do kubectl --context="${ctx}" create -f [PATH/TO/APP/FILES] done
Reserve a static IP
Request a static IP for the multi-cluster Ingress on Google Cloud Platform by running the following command:
gcloud compute addresses create --global [IP_RESOURCE_NAME]
For example, if you name the IP resource my-mci-ip
, it would be:
gcloud compute addresses create --global my-mci-ip
Prepare the Ingress resource
In each cluster, the spec of the
Ingress resource
must have the metadata annotations
field and the backend
field
configured with values for the multi-cluster Ingress.
The following is an example Ingress resource file, mcingress.yaml
:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-foo annotations: kubernetes.io/ingress.global-static-ip-name: my-mci-ip kubernetes.io/ingress.class: gce-multi-cluster spec: backend: serviceName: service-foo servicePort: 80
Add or modify the annotations
:
- Set the value of
kubernetes.io/ingress.global-static-ip-name
to your[IP_RESOURCE_NAME]
. - Set the value of
kubernetes.io/ingress.class
togce-multi-cluster
.
Set a backend
service:
- Set the value of
serviceName
to the name of the default service. - Set the value of
servicePort
to the port of the default service.
Deploy the multi-cluster Ingress
Use kubemci
to create the multi-cluster Ingress with the following command:
kubemci create [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[optional PROJECT_ID] \ --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]For example:
kubemci create my-mci \
--ingress=mcingress.yaml \
--gcp-project=my-gcp-project \
--kubeconfig=~/mcikubeconfig
It outputs a series of messages to confirm the creation process, for example:
Ensuring health checks
...
Health check mci1-hc-30061--my-mci created successfully
...
Backend service mci1-be-30061--my-mci created successfully
...
URL Map mci1-um--my-mci created successfully
...
Target http proxy mci1-tp--my-mci created successfully
...
Forwarding rule mci1-fw--my-mci created successfully
...
Firewall rule mci1-fr--my-mci created successfully
Success.
Getting the status of a multi-cluster Ingress
You can list all the existing multi-cluster Ingresses that were created using kubemci by running the following command:
kubemci list --gcp-project=[PROJECT_ID]
It outputs a list of multi-cluster Ingresses similar the following:
NAME IP CLUSTERS
my-mci 1.2.3.4 gke_my-gcp-project_asia-east1-b_cluster-asia-east, gke_my-gcp-project_europe-west1-c_cluster-eu-west, gke_my-gcp-project_us-east4-a_cluster-us-east
You can then get the status of a specific multi-cluster Ingress with the following command:
kubemci get-status [INGRESS_NAME] --gcp-project=[PROJECT_ID]
It outputs a status message similar to the following:
Load balancer my-mci has IPAddress 1.2.3.4 and is spread across 3 clusters
(gke_project-name_asia-east1-b_cluster-asia-east, gke_project-name_europe-west1-c_cluster-eu-west,
gke_project-name_us-east4-a_cluster-us-east)
Updating an existing multi-cluster Ingress
using --force flag
To update an existing multi-cluster Ingress, run the kubemci create
command with the --force
flag:
kubemci create [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[PROJECT_ID] \ --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE] \ --force
delete, recreate
Alternatively, to update a multi-cluster Ingress, you can delete it and then create it again:
kubemci delete [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[PROJECT_ID] \ --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]
kubemci create [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[PROJECT_ID] \ --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]
Removing clusters from an existing multi-cluster Ingress
To remove clusters from an existing multi-cluster Ingress, first create a
kubeconfig that contains a list of clusters that are to be removed from the
cluster, then run the kubemci remove-clusters
command:
kubemci remove-clusters [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[PROJECT_ID] \ --kubeconfig=[PATH/TO/REMOVAL_KUBECONFIG_FILE]
Deleting a multi-cluster Ingress
To delete a multi-cluster Ingress, use the kubemci delete
command:
kubemci delete [INGRESS_NAME] \ --ingress=[PATH/TO/INGRESS_FILE] \ --gcp-project=[PROJECT_ID] \ --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]
It outputs a series of messages to confirm the deletion process, for example:
Deleting Ingress from cluster: gke_project-name_asia-east1-b_cluster-asia-east...
...
firewall rule mci1-fr--my-mci deleted successfully
...
http forwarding rule mci1-fw--my-mci deleted successfully
...
target HTTPS proxy mci1-tps--my-mci deleted successfully
...
ssl cert mci1-ssl--my-mci deleted successfully
...
url map mci1-um--my-mci deleted successfully
...
Successfully deleted all backend services
...
Successfully deleted all health checks
What's next
- Follow the
kubemci
Zone Printer tutorial to deploy a sample application to multiple zones. - Use the troubleshooting guide to address common issues.
- Read the GKE network overview.
- Learn more about setting up HTTP Load Balancing with Ingress.
- Read Frequently Asked Questions about
kubemci
.