Setting up a multi-cluster Ingress

Beta

This is a Beta release of kubemci. This feature is not covered by any SLA or deprecation policy and might be subject to backward-incompatible changes.

Overview

Caution: The kubemci tool is a temporary solution intended to help users begin using multi-cluster Ingress. This tool will be replaced by an implementation using kubectl that delivers a more Kubernetes-native experience. Once the kubectl implementation is available, you will need to manually migrate any apps that use kubemci.

If you have an application running on multiple Google Kubernetes Engine clusters located in different regions, you can route traffic to a cluster in the region closest to the user.

This page uses the kubemci command-line tool to configure a Cloud HTTP Load Balancer using a Kubernetes Ingress to route traffic across multiple clusters in different regions.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have installed the Cloud SDK.

• Initialize gcloud with the following command:

  gcloud init

  • Choose the account you would like to use or login with a new account.
  • Enter your project ID or create a new project.
  • Configure a default Compute Region and Zone.

• Authenticate and allow Google Cloud SDK to view and manage your data across Google Cloud Platform services:

  gcloud auth login

• Authenticate and allow Google Auth Library to view and manage your data across Google Cloud Platform services:

  gcloud auth application-default login

• Install the kubectl command-line tool:

  gcloud components install kubectl

• Ensure that you have two or more GKE clusters running in the same GCP project and network.

Download the kubemci command-line tool

kubemci is the primary command-line tool used to create and configure a multi-cluster ingress.

Download one of the binaries and place it in an executable path:

• Linux
• macOS

Ensure that the binary is executable. This can be done in the directory of the binary with the following command:

chmod +x ./kubemci

Creating a multi-cluster ingress

Prepare your clusters

Note: A multi-cluster ingress requires clusters with masters running Kubernetes 1.8.1 or higher.

For each Service you are planning to use in the multi-cluster ingress, it must be configured the same across all of the clusters. Specifically, that Service must:

• Have the same name in all of the clusters.
• Be in the same namespace in all of the clusters.
• Be of type NodePort.

• Use the same port number for all of the clusters.

  For example, if your Ingress requires two services, service-foo and service-bar, then:

  • service-foo is a nodePort on port 30016 in all of the clusters.
  • service-bar is a nodePort on port 30017 in all of the clusters.

Refer to the Deploy a nodePort service to each cluster section for more detailed instructions.

Create a kubeconfig file

The kubemci command-line tool requires a kubeconfig file that contains the cluster connection information of the GKE clusters that are going to receive traffic.

Run the following command for each cluster to create a kubeconfig file containing credentials for all of the clusters:

KUBECONFIG=[PATH/TO/MCI_KUBECONFIG_FILE]
gcloud container clusters get-credentials \
    --zone=[ZONE_NAME] [CLUSTER_NAME]

For example:

KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=us-east4-a  cluster-us-east
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=europe-west1-c  cluster-eu-west
KUBECONFIG=~/mcikubeconfig gcloud container clusters get-credentials --zone=asia-east1-b  cluster-apac-east

Deploy a NodePort service to each cluster

Place the Service and application deployment manifest files in the same directory for easy deployment. For example, the following files would both be placed in the app/ directory:

Example Deployment manifest file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 2
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: hello-app
        image: gcr.io/google-samples/hello-app:1.0

Example Service manifest file

apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-app
  name: service-foo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    name: http
    nodePort: 30061
  selector:
    app: my-app
  type: NodePort

Note: For each service that is part of the multi-cluster Ingress, you must manually specify the nodePort in its manifest before creating it in each cluster. If not explicitly specified, Kubernetes assigns a random nodePort by default, which may result in the service having different nodePorts in different clusters.

You can iterate through all of the clusters and get the contexts to deploy the application, along with its nodePort service, by running the following loop:

KUBECONFIG=[PATH/TO/MCI_KUBECONFIG_FILE]
for ctx in $(kubectl config get-contexts -o name); do
  kubectl --context="${ctx}" create -f [PATH/TO/APP/FILES]
done

Reserve a static IP

Request a static IP for the multi-cluster Ingress on Google Cloud Platform by running the following command:

gcloud compute addresses create --global [IP_RESOURCE_NAME]

For example, if you name the IP resource my-mci-ip, it would be:

gcloud compute addresses create --global my-mci-ip

Prepare the Ingress resource

In each cluster, the spec of the Ingress resource must have the metadata annotations field and the backend field configured with values for the multi-cluster Ingress.

The following is an example Ingress resource file, mcingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-foo
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-mci-ip
    kubernetes.io/ingress.class: gce-multi-cluster
spec:
  backend:
    serviceName: service-foo
    servicePort: 80

Add or modify the annotations:

• Set the value of kubernetes.io/ingress.global-static-ip-name to your [IP_RESOURCE_NAME].
• Set the value of kubernetes.io/ingress.class to gce-multi-cluster.

Set a backend service:

• Set the value of serviceName to the name of the default service.
• Set the value of servicePort to the port of the default service.

Deploy the multi-cluster Ingress

Use kubemci to create the multi-cluster Ingress with the following command:

kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[optional PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]

For example:

kubemci create my-mci \
    --ingress=mcingress.yaml \
    --gcp-project=my-gcp-project \
    --kubeconfig=~/mcikubeconfig

It outputs a series of messages to confirm the creation process, for example:

Ensuring health checks
...
Health check mci1-hc-30061--my-mci created successfully
...
Backend service mci1-be-30061--my-mci created successfully
...
URL Map mci1-um--my-mci created successfully
...
Target http proxy mci1-tp--my-mci created successfully
...
Forwarding rule mci1-fw--my-mci created successfully
...
Firewall rule mci1-fr--my-mci created successfully

Success.

Getting the status of a multi-cluster Ingress

You can list all the existing multi-cluster Ingresses that were created using kubemci by running the following command:

kubemci list --gcp-project=[PROJECT_ID]

It outputs a list of multi-cluster Ingresses similar the following:

NAME           IP              CLUSTERS
my-mci   1.2.3.4   gke_my-gcp-project_asia-east1-b_cluster-asia-east, gke_my-gcp-project_europe-west1-c_cluster-eu-west, gke_my-gcp-project_us-east4-a_cluster-us-east

You can then get the status of a specific multi-cluster Ingress with the following command:

kubemci get-status [INGRESS_NAME] --gcp-project=[PROJECT_ID]

It outputs a status message similar to the following:

Load balancer my-mci has IPAddress 1.2.3.4 and is spread across 3 clusters
(gke_project-name_asia-east1-b_cluster-asia-east, gke_project-name_europe-west1-c_cluster-eu-west,
gke_project-name_us-east4-a_cluster-us-east)

Updating an existing multi-cluster Ingress

using --force flag

To update an existing multi-cluster Ingress, run the kubemci create command with the --force flag:

kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE] \
    --force

Note: If you modify your kubeconfig file, running this command with a deployed multi-cluster Ingress only updates the Ingress objects in the clusters specified in the current kubeconfig. Specifically, if contexts are added, the Ingress is spread to the newly added clusters. If contexts are removed, removed clusters will not see the updates.

delete, recreate

Alternatively, to update a multi-cluster Ingress, you can delete it and then create it again:

kubemci delete [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]

kubemci create [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]

Note: You will incur downtime with this method because it takes time for traffic to be forwarded to the newly created Ingress load balancers.

Removing clusters from an existing multi-cluster Ingress

To remove clusters from an existing multi-cluster Ingress, first create a kubeconfig that contains a list of clusters that are to be removed from the cluster, then run the kubemci remove-clusters command:

kubemci remove-clusters [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/REMOVAL_KUBECONFIG_FILE]

Deleting a multi-cluster Ingress

To delete a multi-cluster Ingress, use the kubemci delete command:

kubemci delete [INGRESS_NAME] \
    --ingress=[PATH/TO/INGRESS_FILE] \
    --gcp-project=[PROJECT_ID] \
    --kubeconfig=[PATH/TO/MCI_KUBECONFIG_FILE]

It outputs a series of messages to confirm the deletion process, for example:

Deleting Ingress from cluster: gke_project-name_asia-east1-b_cluster-asia-east...
...
firewall rule mci1-fr--my-mci deleted successfully
...
http forwarding rule mci1-fw--my-mci deleted successfully
...
target HTTPS proxy mci1-tps--my-mci deleted successfully
...
ssl cert mci1-ssl--my-mci deleted successfully
...
url map mci1-um--my-mci deleted successfully
...
Successfully deleted all backend services
...
Successfully deleted all health checks

What's next

• Follow the kubemci Zone Printer tutorial to deploy a sample application to multiple zones.
• Use the troubleshooting guide to address common issues.
• Read the GKE network overview.

• Learn more about:

  • Setting up HTTP Load Balancing with Ingress.
  • Kubernetes Services and Ingress.
  • Kubernetes Engine Multi-Zone and Regional Clusters.
  • kubemci Frequently Asked Questions.