Deploying Ingress across clusters

This page shows you how to deploy an Ingress that serves an application across multiple GKE clusters.

Deployment tutorial

In the tasks below, you will deploy a fictional app named zoneprinter and Multi Cluster Ingress in two clusters. The Ingress provides a shared virtual IP (VIP) address for the app deployments.

This page builds upon the work done in Setting up Multi Cluster Ingress, where you created and registered two clusters. Confirm you have two clusters that are also registered to a fleet:

gcloud container clusters list

The output is similar to the following:

NAME    LOCATION        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
gke-eu  europe-west1-c  1.16.8-gke.9    ***             e2-medium      1.16.8-gke.9     2          RUNNING
gke-us  us-central1-a   1.16.8-gke.9    ***             e2-medium      1.16.6-gke.13 *  2          RUNNING

Creating the Namespace

Because fleets have the property of namespace sameness, we recommend that you coordinate Namespace creation and management across clusters so identical Namespaces are owned and managed by the same group. You can create Namespaces per team, per environment, per application, or per application component. Namespaces can be as granular as necessary, as long as a Namespace ns1 in one cluster has the same meaning and usage as ns1 in another cluster.

In this example, you create a zoneprinter Namespace for each application in each cluster.

  1. Create a file named namespace.yaml that has the following content:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: zoneprinter
    
  2. Switch to the gke-us context:

    kubectl config use-context gke-us
    
  3. Create the Namespace:

    kubectl apply -f namespace.yaml
    
  4. Switch to the gke-eu context:

    kubectl config use-context gke-eu
    
  5. Create the Namespace:

    kubectl apply -f namespace.yaml
    

    The output is similar to the following:

    namespace/zoneprinter created
    

Deploying the app

  1. Create a file named deploy.yaml that has the following content:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: zone-ingress
      namespace: zoneprinter
      labels:
        app: zoneprinter
    spec:
      selector:
        matchLabels:
          app: zoneprinter
      template:
        metadata:
          labels:
            app: zoneprinter
        spec:
          containers:
          - name: frontend
            image: gcr.io/google-samples/zone-printer:0.2
            ports:
            - containerPort: 8080
    
  2. Switch to the gke-us context:

    kubectl config use-context gke-us
    
  3. Deploy the zoneprinter app:

    kubectl apply -f deploy.yaml
    
  4. Switch to the gke-eu context:

    kubectl config use-context gke-eu
    
  5. Deploy the zoneprinter app:

    kubectl apply -f deploy.yaml
    
  6. Verify that the zoneprinter app has successfully deployed in each cluster:

    kubectl get deployment --namespace zoneprinter
    

    The output should be similar to the following in both clusters:

    NAME           READY   UP-TO-DATE   AVAILABLE   AGE
    zone-ingress   1/1     1            1           12m
    

Deploying through the config cluster

Now that the application is deployed across gke-us and gke-eu, you will deploy a load balancer by deploying MultiClusterIngress (MCI) and MultiClusterService (MCS) resources in the config cluster. MCI and MCS are custom resources (CRDs) that are the multi-cluster equivalents of Ingress and Service resources.

In the setup guide, you configured the gke-us cluster as the config cluster. The config cluster is used to deploy and configure Ingress across all clusters.

  1. Set the context to the config cluster.

    kubectl config use-context gke-us
    

MultiClusterService

  1. Create a file named mcs.yaml that has the following content:

    apiVersion: networking.gke.io/v1
    kind: MultiClusterService
    metadata:
      name: zone-mcs
      namespace: zoneprinter
    spec:
      template:
        spec:
          selector:
            app: zoneprinter
          ports:
          - name: web
            protocol: TCP
            port: 8080
            targetPort: 8080
    
  2. Deploy the MultiClusterService resource that matches the zoneprinter app:

    kubectl apply -f mcs.yaml
    
  3. Verify that the zone-mcs resource has successfully deployed in the config cluster:

    kubectl get mcs -n zoneprinter
    

    The output is similar to the following:

    NAME       AGE
    zone-mcs   9m26s
    

    This MCS creates a derived headless Service in every cluster that matches Pods with app: zoneprinter. You can see that one exists in the gke-us cluster kubectl get service -n zoneprinter

    The output is similar to the following:

    NAME                                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
    mci-zone-mcs-svc-lgq966x5mxwwvvum   ClusterIP   None          <none>        8080/TCP         4m59s
    

A similar headless Service will also exist in gke-eu. These local Services are used to dynamically select Pod endpoints to program the global Ingress load balancer with backends.

MultiClusterIngress

  1. Create a file named mci.yaml that has the following content:

    apiVersion: networking.gke.io/v1
    kind: MultiClusterIngress
    metadata:
      name: zone-ingress
      namespace: zoneprinter
    spec:
      template:
        spec:
          backend:
            serviceName: zone-mcs
            servicePort: 8080
    

    Note that this configuration routes all traffic to the MultiClusterService named zone-mcs that exists in the zoneprinter namespace.

  2. Deploy the MultiClusterIngress resource that references zone-mcs as a backend:

    kubectl apply -f mci.yaml
    

    The output is similar to the following:

    multiclusteringress.networking.gke.io/zone-mci created
    

    Note that MultiClusterIngress has the same schema as the Kubernetes Ingress. The Ingress resource semantics are also the same with the exception of the backend.serviceName field.

The backend.serviceName field in a MultiClusterIngress references a MultiClusterService in the fleet API rather than a Service in a Kubernetes cluster. This means that any of the settings for Ingress, such as TLS termination, settings can be configured in the same way.

Validating a successful deployment status

Google Cloud Load Balancer deployment may take several minutes to deploy for new load balancers. Updating existing load balancers completes faster because new resources do not need to be deployed. The MCI resource details the underlying Compute Engine resources that have been created on behalf of the MCI.

  1. Verify that deployment has succeeded:

    kubectl describe mci zone-ingress -n zoneprinter
    

    The output is similar to the following:

    Name:         zone-ingress
    Namespace:    zoneprinter
    Labels:       <none>
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"networking.gke.io/v1","kind":"MultiClusterIngress","metadata":{"annotations":{},"name":"zone-ingress","namespace":"zon...
    API Version:  networking.gke.io/v1
    Kind:         MultiClusterIngress
    Metadata:
      Creation Timestamp:  2020-04-10T23:35:10Z
      Finalizers:
        mci.finalizer.networking.gke.io
      Generation:        2
      Resource Version:  26458887
      Self Link:         /apis/networking.gke.io/v1/namespaces/zoneprinter/multiclusteringresses/zone-ingress
      UID:               62bec0a4-8a08-4cd8-86b2-d60bc2bda63d
    Spec:
      Template:
        Spec:
          Backend:
            Service Name:  zone-mcs
            Service Port:  8080
    Status:
      Cloud Resources:
        Backend Services:
          mci-8se3df-8080-zoneprinter-zone-mcs
        Firewalls:
          mci-8se3df-default-l7
        Forwarding Rules:
          mci-8se3df-fw-zoneprinter-zone-ingress
        Health Checks:
          mci-8se3df-8080-zoneprinter-zone-mcs
        Network Endpoint Groups:
          zones/europe-west1-c/networkEndpointGroups/k8s1-e4adffe6-zoneprint-mci-zone-mcs-svc-lgq966x5m-808-88670678
          zones/us-central1-a/networkEndpointGroups/k8s1-a6b112b6-zoneprint-mci-zone-mcs-svc-lgq966x5m-808-609ab6c6
        Target Proxies:
          mci-8se3df-zoneprinter-zone-ingress
        URL Map:  mci-8se3df-zoneprinter-zone-ingress
      VIP:        34.98.102.37
    Events:
      Type    Reason  Age                    From                              Message
      ----    ------  ----                   ----                              -------
      Normal  ADD     3m35s                  multi-cluster-ingress-controller  zoneprinter/zone-ingress
      Normal  UPDATE  3m10s (x2 over 3m34s)  multi-cluster-ingress-controller  zoneprinter/zone-ingress
    

There are several fields that indicate the status of this Ingress deployment:

  • Cloud Resourcelists the Compute Engine resources like forwarding rules, backend services, and firewall rules that have been created by the Multi Cluster Ingress controller. If these are not listed it means that they have not been created yet. You can inspect individual Compute Engine resources with the Console or gcloud command to get its status.

  • VIP lists an IP address when one has been allocated. Note that the load balancer may not yet be processing traffic even though the VIP exists. If you do not see a VIP after a couple minutes, or if the load balancer is not serving a 200 response within 10 minutes, see Troubleshooting and operations.

  • Events is the first place to look. If an error has occurred it will be listed here.

If the output events are Normal, then the MCI deployment is likely successful, but the only way to determine that the full traffic path is functional is to test it.

  1. Validate that the application is serving on the VIP with the /ping endpoint:

    curl INGRESS_VIP/ping
    

    Replace INGRESS_VIP with the virtual IP (VIP) address.

    The output is similar to the following:

    {"Hostname":"34.98.102.37","Version":"1.0","GCPZone":"us-central1-a","Backend":"zone-ingress-5554bb95b4-svv5d"}
    

    The output should indicate the region and backend of the application.

  2. You can also go to the http://INGRESS_VIP URL in your browser to see a graphical version of the application that shows the region that it's being served from.

    The cluster that the traffic is forwarded to depends on your location. The GCLB is designed to forward client traffic to the closest available backend with capacity.

Resource specs

MultiClusterService spec

The MultiClusterService definition consists of two pieces:

  1. A template section that defines the Service to be created in the Kubernetes clusters. Note that while the template section contains fields supported in a typical Service, there are only two fields that are supported in a MultiClusterService: selector and ports. The other fields are ignored.

  2. An optional clusters section that defines which clusters receive traffic and the load balancing properties for each cluster. If no clusters section is specified or if no clusters are listed, all clusters are used by default.

The following manifest describes a standard MCS:

apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
  name: NAME
  namespace: NAMESPACE
spec:
  template:
    spec:
      selector:
        app: POD_LABEL
      ports:
      - name: web
        protocol: TCP
        port: PORT
        targetPort: TARGET_PORT

Replace the following:

  • NAME: the name of the MCS. This name is referenced by the serviceName field in the MCI resources.
  • NAMESPACE: the Kubernetes Namespace that the MCS is deployed in. It must match be in the same Namespace as the MCI and the Pods across all clusters in the fleet.
  • POD_LABEL: the label that determines which pods are selected as backends for this MCS across all clusters in the fleet.
  • PORT must match with the port referenced by the MCI that references this MCS.
  • TARGET_PORT: the port that is used to send traffic to the Pod from the GCLB. A NEG is created in each cluster with this port as its serving port.

MultiClusterIngress spec

The following mci.yaml describes the load balancer frontend:

apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: NAME
  namespace: NAMESPACE
spec:
  template:
    spec:
      backend:
       serviceName: DEFAULT_SERVICE
       servicePort: PORT
      rules:
        - host: HOST_HEADER
          http:
            paths:
            - path: PATH
              backend:
                serviceName: SERVICE
                servicePort: PORT

Replace the following:

  • NAME: the name of the MCI resource.
  • NAMESPACE: the Kubernetes Namespace that the MCI is deployed in. It must be in the same Namespace as the MCS and the Pods across all clusters in the fleet.
  • DEFAULT_SERVICE: acts as the default backend for all traffic that does not match any host or path rules. This is a required field and a default backend must be specified in the MCI even if there are other host or path matches configured.
  • PORT: any valid port number. This must match with the port field of the MCS resources.
  • HOST_HEADER: matches traffic by the HTTP host header field. The host field is optional.
  • PATH: matches traffic by the path of the HTTP URL. The path field is optional.
  • SERVICE: the name of an MCS that is deployed in the same Namespace and config cluster as this MCI.

Multi Cluster Ingress features

This section shows you how to configure additional Multi Cluster Ingress features.

Cluster selection

By default, Services derived from Multi Cluster Ingress are scheduled on every member cluster. However, you may want to apply ingress rules to specific clusters. Some use-cases include:

  • Applying Multi Cluster Ingress to all clusters but the config cluster for isolation of the config cluster.
  • Migrating workloads between clusters in a blue-green fashion.
  • Routing to application backends that only exist in a subset of clusters.
  • Using a single L7 VIP for host/path routing to backends that live on different clusters.

Cluster selection allows you to select clusters by region/name in the MultiClusterService object. This controls which clusters your Multi Cluster Ingress is pointing to and where the derived Services are scheduled. Clusters within the same fleet and region should not have the same name so that clusters can be referenced uniquely.

  1. Open mcs.yaml

    apiVersion: networking.gke.io/v1
    kind: MultiClusterService
    metadata:
      name: zone-mcs
      namespace: zoneprinter
    spec:
      template:
        spec:
          selector:
            app: zoneprinter
          ports:
          - name: web
            protocol: TCP
            port: 8080
            targetPort: 8080
    

    This specification currently creates Derived Services in all clusters, the default behavior.

  2. Append the following lines in the clusters section:

    apiVersion: networking.gke.io/v1
    kind: MultiClusterService
    metadata:
      name: zone-mcs
      namespace: zoneprinter
    spec:
      template:
        spec:
          selector:
            app: zoneprinter
          ports:
          - name: web
            protocol: TCP
            port: 8080
            targetPort: 8080
      clusters:
      - link: "us-central1-a/gke-us"
      - link: "europe-west1-c/gke-eu"
    

    This example creates Derived Service resources only in gke-us and gke-eu clusters. You must select clusters to selectively apply ingress rules. If the "clusters" section of the MultiClusterService is not specified or if no clusters are listed, it is interpreted as the default "all" clusters.

HTTPS support

The Kubernetes Secret supports HTTPS. Before enabling HTTPS support, you must create a static IP address. This static IP allows HTTP and HTTPS to share the same IP address. For more information, see Creating a static IP.

Once you have created a static IP address, you can create a Secret.

  1. Create a Secret:

    kubectl -n prod create secret tls SECRET_NAME --key PATH_TO_KEYFILE --cert PATH_TO_CERTFILE
    

    Replace the following:

    • SECRET_NAME with the name of your Secret.
    • PATH_TO_KEYFILE with the path to the TLS key file.
    • PATH_TO_CERTFILE with the path to the TLS certificate file.
  2. Update the mci.yaml file with the static IP and Secret:

    apiVersion: networking.gke.io/v1
    kind: MultiClusterIngress
    metadata:
      name: zone-ingress
      namespace: zoneprinter
      annotations:
        networking.gke.io/static-ip: STATIC_IP_ADDRESS
    spec:
      template:
        spec:
          backend:
            serviceName: zone-mcs
            servicePort: 8080
          tls:
          - secretName: SECRET_NAME
    

    Replace the following:

    • STATIC_IP_ADDRESS: the actual IP address, not the resource name.
    • SECRET_NAME: the name of your Secret.

BackendConfig support

The BackendConfig CRD allows you to customize settings on the Compute Engine BackendService resource. The supported specification is below:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: zone-health-check-cfg
  namespace: zoneprinter
spec:
  healthCheck:
    checkIntervalSec: [int]
    timeoutSec: [int]
    healthyThreshold: [int]
    unhealthyThreshold: [int]
    type: [HTTP | HTTPS | HTTP2 | TCP]
    port: [int]
    requestPath: [string]
  timeoutSec: [int]
  connectionDraining:
    drainingTimeoutSec: [int]
  sessionAffinity:
    affinityType: [CLIENT_IP | CLIENT_IP_PORT_PROTO | CLIENT_IP_PROTO | GENERATED_COOKIE | HEADER_FIELD | HTTP_COOKIE | NONE]
    affinityCookieTtlSec: [int]
  cdn:
    enabled: [bool]
    cachePolicy:
      includeHost: [bool]
      includeQueryString: [bool]
      includeProtocol: [bool]
      queryStringBlacklist: [string list]
      queryStringWhitelist: [string list]
  securityPolicy:
    name: ca-how-to-security-policy
  logging:
    enable: [bool]
    sampleRate: [float]
  iap:
    enabled: [bool]
    oauthclientCredentials:
      secretName: [string]

To use BackendConfig, attach it on your MultiClusterService resource using an annotation:

apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
  name: zone-mcs
  namespace: zoneprinter
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {"8080":"zone-health-check-cfg"}}'
spec:
 template:
   spec:
     selector:
       app: zoneprinter
     ports:
     - name: web
       protocol: TCP
       port: 8080
       targetPort: 8080

For more information about BackendConfig semantics, see Associating a service port with a BackendConfig.

gRPC support

Configuring gRPC applications on Multi Cluster Ingress requires very specific setup. Here are some tips to make sure your load balancer is configured properly:

  1. Make sure that the traffic from the load balancer to your application is HTTP/2. Use application protocols to configure this.
  2. Make sure that your application is properly configured for SSL since this is a requirement of HTTP/2. Note that using self-signed certs is acceptable.
  3. You must turn off mTLS on your application because mTLS is not supported for L7 external load balancers.

Resource lifecycle

Configuration changes

MultiClusterIngress and MultiClusterService resources behave as standard Kubernetes objects, so changes to the objects are asynchronously reflected in the system. Any changes that result in an invalid configuration cause associated Google Cloud objects to remain unchanged and raise an error in the object event stream. Errors associated with the configuration will be reported as events.

Managing Kubernetes resources

Deleting the Ingress object tears down the HTTP(S) load balancer so traffic is no longer forwarded to any defined MultiClusterService.

Deleting the MultiClusterService removes the associated derived services in each of the clusters.

Managing clusters

The set of clusters targeted by the load balancer can be changed by adding or removing a Membership.

For example, to remove the gke-eu cluster as a backend for an ingress, run:

gcloud container hub memberships unregister CLUSTER_NAME \
  --gke-uri=URI

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • URI: the URI of the GKE cluster.

To add a cluster in Europe, run:

gcloud container hub memberships register europe-cluster \
  --context=europe-cluster --service-account-key-file=PATH_TO_SERVICE_ACCOUNT_KEY_FILE

Replace the PATH_TO_SERVICE_ACCOUNT_KEY_FILE with the path to the key file of the Service account.

Note that registering or unregistering a cluster changes its status as a backend for all Ingresses. In the above case, unregistering the gke-eu cluster removes it as an available backend for all Ingresses you create. The reverse is true for registering a new cluster.

Disabling Multi Cluster Ingress

Before disabling Multi Cluster Ingress you must ensure that you first delete your MultiClusterIngress and MultiClusterService resources and verify any associated networking resources are deleted.

Then, to disable Multi Cluster Ingress, use the following command:

gcloud alpha container hub ingress disable

If you do not delete MultiClusterIngress and MultiClusterService resources before disabling Multi Cluster Ingress, you might encounter an error similar to the following:

Feature has associated resources that should be cleaned up before deletion.

If you want to force disable Multi Cluster Ingress, use the following command:

gcloud alpha container hub ingress disable --force

Annotations

The following annotations are supported on MultiClusterIngress and MultiClusterService resources.

MultiClusterIngress Annotations

Annotation Description
networking.gke.io/frontend-config References a FrontendConfig resource in the same Namespace as the MultiClusterIngress resource.
networking.gke.io/static-ip Refers to the literal IP address of a global static IP.
networking.gke.io/pre-shared-certs Refers to a global SSLCertificate resource.

MultiClusterService Annotations

Annotation Description
networking.gke.io/app-protocols Use this annotation to set the protocol for communication between the load balancer and the application. Possible protocols are HTTP, HTTPS, and HTTP/2. See HTTPS between load balancer and your application and HTTP/2 for load balancing with Ingress.
beta.cloud.google.com/backend-config Use this annotation to configure the backend service associated with a servicePort. For more information, see Ingress features.

SSL Policies and HTTPS Redirects

You can use the FrontendConfig resource to configure SSL policies and HTTPS redirects. SSL policies allow you to specify which cipher suites and TLS versions are accepted by the load balancer. HTTPS redirects allow you to enforce the redirection from HTTP/port 80 to HTTPS/port 443. The following steps configure an SSL policy and HTTPS redirect together. Note that they can also be configured independently.

  1. Create an SSL policy that will reject requests using a version lower than TLS v1.2.

    gcloud compute ssl-policies create tls-12-policy \
     --profile MODERN \
     --min-tls-version 1.2 \
     --project=PROJECT_ID
    

    Replace PROJECT_ID with the project ID where your GKE clusters are running.

  2. View your policy to ensure it has been created.

    gcloud compute ssl-policies list --project=PROJECT_ID
    

    The output is similar to the following:

    NAME           PROFILE  MIN_TLS_VERSION
    tls-12-policy  MODERN   TLS_1_2
    
  3. Create a certificate for foo.example.com as in this example. Once you have the key.pem and cert.pem, store these credentials as a Secret that will be referenced by the MultiClusterIngress resource.

    kubectl -n multi-cluster-demo create secret tls SECRET_NAME --key key.pem --cert cert.pem
    
  4. Save the following FrontendConfig resource as frontendconfig.yaml. See Configuring FrontendConfig resources for more information on the supported fields within a FrontendConfig.

    apiVersion: networking.gke.io/v1beta1
    kind: FrontendConfig
    metadata:
      name: frontend-redirect-tls-policy
      namespace: multi-cluster-demo
    spec:
      sslPolicy: tls-12-policy
      redirectToHttps:
        enabled: true
    

    This FrontendConfig will enable HTTPS redirects and an SSL policy that enforces a minimum TLS version of 1.2.

  5. Deploy frontendconfig.yaml into your config cluster.

    kubectl apply -f frontendconfig.yaml --context MCI_CONFIG_CLUSTER
    

    Replace the MCI_CONFIG_CLUSTER with the name of your config cluster.

  6. Save the following MultiClusterIngress as mci-frontendconfig.yaml.

    apiVersion: networking.gke.io/v1
    kind: MultiClusterIngress
    metadata:
      name: foo-ingress
      namespace: multi-cluster-demo
      annotations:
        networking.gke.io/frontend-config: frontend-redirect-tls-policy
        networking.gke.io/static-ip: STATIC_IP_ADDRESS
    spec:
      template:
        spec:
          backend:
            serviceName: default-backend
            servicePort: 8080
          rules:
          - host: foo.example.com
            http:
              paths:
                - backend:
                    serviceName: foo
                    servicePort: 8080
          tls:
          - secretName: SECRET_NAME
    
    • Replace STATIC_IP_ADDRESS with a static global IP address that you have already provisioned.
    • Replace SECRET_NAME with the Secret where your foo.example.com certificate is stored.

    There are two requirements when enabling HTTPS redirects:

    • TLS must be enabled, either through the spec.tls field or through the pre-shared certificate annotation networking.gke.io/pre-shared-certs. The MultiClusterIngress will not deploy if HTTPS redirects is enabled but HTTPS is not.
    • A static IP must be referenced through the networking.gke.io/static-ip annotation. Static IPs are required when enabling HTTPS on a MultiClusterIngress.
  7. Deploy the MultiClusterIngress to your config cluster.

    kubectl apply -f mci-frontendconfig.yaml --context MCI_CONFIG_CLUSTER
    
  8. Wait a minute or two and inspect foo-ingress.

    kubectl describe mci foo-ingress --context MCI_CONFIG_CLUSTER
    

    A successful output resembles the following:

    • The Cloud Resources status is populated with resource names
    • The VIP field is populated with the load balancer IP address
    Name:         foobar-ingress
    Namespace:    multi-cluster-demo
    
    ...
    
    Status:
      Cloud Resources:
        Backend Services:
          mci-otn9zt-8080-multi-cluster-demo-bar
          mci-otn9zt-8080-multi-cluster-demo-default-backend
          mci-otn9zt-8080-multi-cluster-demo-foo
        Firewalls:
          mci-otn9zt-default-l7
        Forwarding Rules:
          mci-otn9zt-fw-multi-cluster-demo-foobar-ingress
          mci-otn9zt-fws-multi-cluster-demo-foobar-ingress
        Health Checks:
          mci-otn9zt-8080-multi-cluster-demo-bar
          mci-otn9zt-8080-multi-cluster-demo-default-backend
          mci-otn9zt-8080-multi-cluster-demo-foo
        Network Endpoint Groups:
          zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluste-mci-default-backend-svc--80-9e362e3d
          zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluster--mci-bar-svc-067a3lzs8-808-89846515
          zones/europe-west1-b/networkEndpointGroups/k8s1-1869d397-multi-cluster--mci-foo-svc-820zw3izx-808-8bbcb1de
          zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluste-mci-default-backend-svc--80-a528cc75
          zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluster--mci-bar-svc-067a3lzs8-808-36281739
          zones/us-central1-b/networkEndpointGroups/k8s1-a63e24a6-multi-cluster--mci-foo-svc-820zw3izx-808-ac733579
        Target Proxies:
          mci-otn9zt-multi-cluster-demo-foobar-ingress
          mci-otn9zt-multi-cluster-demo-foobar-ingress
        URL Map:  mci-otn9zt-rm-multi-cluster-demo-foobar-ingress
      VIP:        34.149.29.76
    Events:
      Type     Reason  Age                From                              Message
      ----     ------  ----               ----                              -------
      Normal   UPDATE  38m (x5 over 62m)  multi-cluster-ingress-controller  multi-cluster-demo/foobar-ingress
    
  9. Verify that HTTPS redirects function correctly by sending an HTTP request through curl.

    curl VIP
    

    Replace VIP with the MultiClusterIngress IP address.

    The output should show that the request was redirected to the HTTPS port which indicates that redirects are functioning correctly.

  10. Verify that the TLS policy functions correctly by sending an HTTPS request using TLS version 1.1. Because DNS is not configured for this domain, use the --resolve option to tell curl to resolve the IP address directly.

    curl https://foo.example.com --resolve foo.example.com:443:VIP --cacert CERT_FILE -v
    

    This step requires the certificate PEM file used to secure the MultiClusterIngress. A successful output will look similar to the following:

    ...
    * SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
    * ALPN, server accepted to use h2
    * Server certificate:
    *  subject: O=example; CN=foo.example.com
    *  start date: Sep  1 10:32:03 2021 GMT
    *  expire date: Aug 27 10:32:03 2022 GMT
    *  common name: foo.example.com (matched)
    *  issuer: O=example; CN=foo.example.com
    *  SSL certificate verify ok.
    * Using HTTP2, server supports multi-use
    * Connection state changed (HTTP/2 confirmed)
    * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    * Using Stream ID: 1 (easy handle 0x7fa10f00e400)
    > GET / HTTP/2
    > Host: foo.example.com
    > User-Agent: curl/7.64.1
    > Accept: */*
    >
    * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
    < HTTP/2 200
    < content-type: application/json
    < content-length: 308
    < access-control-allow-origin: *
    < server: Werkzeug/1.0.1 Python/3.8.6
    < date: Wed, 01 Sep 2021 11:39:06 GMT
    < via: 1.1 google
    < alt-svc: clear
    <
    {"cluster_name":"gke-us","host_header":"foo.example.com","metadata":"foo","node_name":"gke-gke-us-default-pool-22cb07b1-r5r0.c.mark-church-project.internal","pod_name":"foo-75ccd9c96d-dkg8t","pod_name_emoji":"👞","project_id":"mark-church-project","timestamp":"2021-09-01T11:39:06","zone":"us-central1-b"}
    * Connection #0 to host foo.example.com left intact
    * Closing connection 0
    

    The response code is 200 and TLSv1.2 is being used which indicates that everything is functioning properly.

    Next you can verify that the SSL policy enforces the correct TLS version by attempting to connect with TLS 1.1. Your SSL policy must be configured for a minimum version of 1.2 for this step to work.

  11. Send the same request from the previous step, but enforce a TLS version of 1.1.

    curl https://foo.example.com --resolve foo.example.com:443:VIP -v \
      --cacert CERT_FILE \
      --tls-max 1.1
    

    A successful output will look similar to the following:

    * Added foo.example.com:443:34.149.29.76 to DNS cache
    * Hostname foo.example.com was found in DNS cache
    *   Trying 34.149.29.76...
    * TCP_NODELAY set
    * Connected to foo.example.com (34.149.29.76) port 443 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * successfully set certificate verify locations:
    *   CAfile: cert.pem
      CApath: none
    * TLSv1.1 (OUT), TLS handshake, Client hello (1):
    * TLSv1.1 (IN), TLS alert, protocol version (582):
    * error:1400442E:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert protocol version
    * Closing connection 0
    curl: (35) error:1400442E:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert protocol version
    

    The failure to complete the TLS handshake indicates that the SSL policy has blocked TLS 1.1 successfully.

Creating a static IP

  1. Allocate a static IP:

    gcloud compute addresses create ADDRESS_NAME --global
    

    Replace ADDRESS_NAME with the name of the address to create.

    The output is similar to the following:

    Created [https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/addresses/ADDRESS_NAME].
    
  2. View the IP address you just created:

    gcloud compute addresses list
    

    The output is similar to the following:

    NAME          ADDRESS/RANGE  TYPE      STATUS
    ADDRESS_NAME  34.102.201.47  EXTERNAL  RESERVED
    

    This output includes the ADDRESS_NAME you created.

  3. Apply the static IP by updating mci.yaml:

    kubectl get mci zone-mci -o yaml
    

    The output is similar to the following:

    kind: MultiClusterIngress
    metadata:
      name: shopping-service
      namespace: prod
      annotations:
        networking.gke.io/static-ip: STATIC_IP_ADDRESS
    spec:
      template:
        spec:
           backend:
             serviceName: shopping-service
              servicePort: 80
    

    The STATIC_IP_ADDRESS is the actual IP address (34.102.201.47), not the resource name (ADDRESS_NAME).

Pre-shared certificates

Pre-shared certificates are certificates uploaded to Google Cloud that can be used by the load balancer for TLS termination instead of certificates stored in Kubernetes Secrets. These certificates are uploaded out of band from GKE to Google Cloud and referenced by an Multi Cluster Ingress object. Multiple certificates, either through pre-shared certs or Kubernetes secrets, are also supported.

Using the certificates in Multi Cluster Ingress requires the networking.gke.io/pre-shared-certs annotation and the names of the certs. When multiple certificates are specified for a given Multi Cluster Ingress object, a predetermined order governs which cert is presented to the client.

You can list the available SSL certificates by running:

gcloud compute ssl-certificates list

The example below describes client traffic to one of the specified hosts that matches the Common Name of the pre-shared certs so the respective certificate that matches the domain name will be presented.

kind: MultiClusterIngress
metadata:
  name: shopping-service
  namespace: prod
  annotations:
    networking.gke.io/pre-shared-certs: "domain1-cert, domain2-cert"
spec:
  template:
    spec:
      rules:
      - host: my-domain1.gcp.com
        http:
          paths:
          - backend:
              serviceName: domain1-svc
              servicePort: 443
      - host: my-domain2.gcp.com
        http:
          paths:
          - backend:
              serviceName: domain2-svc
              servicePort: 443

Google-managed Certificates

Google-managed Certificates are supported on Multi Cluster Ingress through the networking.gke.io/pre-shared-certs annotation. Multi Cluster Ingress supports the attachment of Google-managed certificates to a MultiClusterIngress resource, however unlike single-cluster Ingress, the declarative generation of a Kubernetes ManagedCertificate resource is not supported on Multi Cluster Ingress. The original creation of the Google-managed certificate must be done directly through the compute ssl-certificates create API before you can attach it to a MultiClusterIngress. That can be done following these steps:

  1. Create a Google-managed Certificate as in step 1 here. Do not move to step 2 as Multi Cluster Ingress will attach this certificate for you.

    gcloud compute ssl-certificates create my-google-managed-cert \
        --domains=my-domain.gcp.com \
        --global
    
  2. Reference the name of the certificate in your MultiClusterIngress using the networking.gke.io/pre-shared-certs annotation.

    kind: MultiClusterIngress
    metadata:
    name: shopping-service
    namespace: prod
    annotations:
      networking.gke.io/pre-shared-certs: "my-google-managed-cert"
    spec:
    template:
      spec:
        rules:
        - host: my-domain.gcp.com
          http:
            paths:
            - backend:
                serviceName: my-domain-svc
                servicePort: 8080
    

The preceding manifest attaches the certificate to your MultiClusterIngress so that it can terminate traffic for your backend GKE clusters. Google Cloud will automatically renew your certificate prior to certificate expiry. Renewals occur transparently and does not require any updates to Multi Cluster Ingress.

Application protocols

The connection from the load balancer proxy to your application uses HTTP by default. Using networking.gke.io/app-protocols annotation, you can configure the load balancer to use HTTPS or HTTP/2 when it forwards requests to your application. In the annotation field of the following example, http2 refers to the MultiClusterService port name and HTTP2 refers to the protocol that the load balancer uses.

kind: MultiClusterService
metadata:
  name: shopping-service
  namespace: prod
  annotations:
    networking.gke.io/app-protocols: '{"http2":"HTTP2"}'
spec:
  template:
    spec:
      ports:
      - port: 443
        name: http2

BackendConfig

Refer to the section above on how to configure the annotation.

Known issues, product limits, and guidance

The following are important limitations or notices about Multi Cluster Ingress behavior that dictate safe and acceptable usage. Contact your account team or gke-mci-feedback@google.com if you have questions.

  • Compute Engine load balancer resources are created with a name containing a prefix of mci-[6 char hash]. All Multi Cluster Ingress managed resources in a project use the same prefix. This prefix is used to manage and garbage collect Compute Engine resources Multi Cluster Ingress deploys. Since this prefix contains a generated hash, it is unlikely that a project will contain Compute Engine resources outside the realm of Multi Cluster Ingress with this same prefix. However, Compute Engine load balancers in the same project that are not managed by Multi Cluster Ingress should not use this prefix or else they will be deleted.

  • Multi Cluster Ingress only supports clusters within the same project. Only one instance of Multi Cluster Ingress can be deployed per project, though the scoping of clusters can be controlled with cluster selection. This allows MultiClusterService resources to be deployed only to specific subsets of clusters within a project.

  • Multi Cluster Ingress and fleets have a pre-configured quota of maximum 15 clusters per project. This quota can be increased if necessary. Contact your account team to request a higher per-project cluster limit if you have requirements for registering more clusters.

  • Multi Cluster Ingress has a pre-configured quota of 50 MultiClusterIngress and 100 MultiClusterService resources per project. This allows up to 50 MCI and 100 MCS resources to be created in a config cluster for any number of backend clusters up to the per-project cluster maximum. This quota can be increased if necessary. Contact your account team to request higher per-project MCI & MCS quotas if you have requirements for higher scale.

  • Configuration of HTTPS requires a pre-allocated static IP address. HTTPS is not currently supported with ephemeral IP addresses.

What's next