Deploying multi-cluster Gateways

This page describes how to deploy Kubernetes Gateway resources for load balancing across multiple Google Kubernetes Engine (GKE) clusters. To see how Gateways are deployed for just a single GKE cluster, see Deploying Gateways.

Multi-cluster Gateways

A multi-cluster Gateway is a Gateway resource that load balances traffic across multiple Kubernetes clusters. In GKE the gke-l7-gxlb-mc and gke-l7-rilb-mc GatewayClasses deploy multi-cluster Gateways that provide HTTP routing, traffic splitting, traffic mirroring, health-based failover, and more across different GKE clusters, Kubernetes Namespaces, and across different regions. Multi-cluster Gateways make managing application networking across many clusters and teams easy, secure, and scalable for infrastructure administrators.

A multi-cluster Gateway is a Gateway resource that load balances traffic
across multiple Kubernetes clusters.

This page introduces three examples to teach you how to deploy multi-cluster Gateways using the GKE Gateway controller:

  • Example 1: An external, multi-cluster Gateway providing load balancing across two GKE clusters for internet traffic.
  • Example 2: Blue-green, weight-based traffic splitting and traffic mirroring across two GKE clusters for internal VPC traffic.

Each of the examples will use the same store and site applications to model a real-world scenario where an online shopping service and a website service are owned and operated by separate teams and deployed across a fleet of shared GKE clusters. Each of the examples highlights different topologies and use cases enabled by multi-cluster Gateways.

Multi-cluster Gateways require some environmental preparation before they can be deployed. Before proceeding, follow the steps in Enabling multi-cluster Gateways:

  1. Deploy GKE clusters.

  2. Register your clusters to a fleet.

  3. Enable the multi-cluster Service and multi-cluster Gateway controllers.

Lastly, review the GKE Gateway controller Preview limitations and known issues before using it in your environment.

Multi-cluster, multi-region, external Gateway

In this tutorial, you will create an external multi-cluster Gateway that serves external traffic across an application running in two GKE clusters.

store.example.com is deployed across two GKE clusters and exposed to the internet via a multi-cluster Gateway

In the following steps you:

  1. Deploy the sample store application to the gke-west-1 and gke-east-1 clusters.
  2. Configure ServiceExport resources on each cluster to export the Services into your fleet.
  3. Deploy a gke-l7-gxlb-mc Gateway and HTTPRoute to your config cluster, gke-west-1.

After the application and Gateway resources are deployed, you can control traffic across the two GKE clusters using path-based routing:

  • Requests to /west are routed to store Pods in the gke-west-1 cluster.
  • Requests to /east are routed to store Pods in thegke-east-1 cluster.
  • Requests to any other path are routed to either cluster, according to its health, capacity, and proximity to the requesting client.

Deploying the demo application

  1. Create the store Deployment and Namespace in all three of the clusters that were deployed in Enabling multi-cluster Gateways:

    kubectl apply --context gke-west-1 -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
    kubectl apply --context gke-west-2 -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
    kubectl apply --context gke-east-1 -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
    

    It deploys the following resources to each cluster:

    namespace/store created
    deployment.apps/store created
    

    All examples in this page use the app deployed in this step. Make sure that the app is deployed across all three clusters before trying any of the remaining steps. This example uses only clusters gke-west-1 and gke-east-1 , and gke-west-2 is used in another example.

Multi-cluster Services

Services are how Pods are exposed to clients. Because the GKE Gateway controller uses container-native load balancing, it does not use the ClusterIP or Kubernetes load balancing to reach Pods. Traffic is sent directly from the load balancer to the Pod IPs. However, Services still play a critical role as a logical identifier for Pod grouping.

Multi-cluster Services (MCS) is both an API standard for Services that span clusters and a GKE controller that provides service discovery across GKE clusters. The multi-cluster Gateway controller uses MCS API resources to group Pods into a Service that is addressable across or spans multiple custers.

The multi-cluster Services API defines the following custom resources:

  • ServiceExports map to a Kubernetes Service, exporting the endpoints of that Service to all clusters registered to the fleet. When a Service has a corresponding ServiceExport it means that the Service can be addressed by a multi-cluster Gateway.
  • ServiceImports are automatically generated by the multi-cluster Service controller. ServiceExport and ServiceImports come in pairs. If a ServiceExport exists in the fleet, then a corresponding ServiceImport is created to allow the Service mapped to the ServiceExport to be accessed from across clusters.

Exporting Services works in the following way. A store Service exists in gke-1 which selects a group of Pods in that cluster. A ServiceExport is created in the cluster which allows the Pods in gke-1 to become accessible from the other clusters in the fleet. The ServiceExport will map to and expose Services that have the same name and Namespace as the ServiceExport resource.

apiVersion: v1
kind: Service
metadata:
  name: store
  namespace: store
spec:
  selector:
    app: store
  ports:
  - port: 8080
    targetPort: 8080
---
kind: ServiceExport
apiVersion: net.gke.io/v1
metadata:
  name: store
  namespace: store

The following diagram shows what happens after a ServiceExport is deployed. If a ServiceExport and Service pair exist then the multi-cluster Service controller deploys a corresponding ServiceImport to every GKE cluster in the fleet. The ServiceImport is the local representation of the store Service in every cluster. This enables the client Pod in gke-2 to use ClusterIP or headless Services to reach the store Pods in gke-1. When used in this manner multi-cluster Services provide east-west load balancing between clusters. To use multi-cluster Services for cluster-to-cluster load balancing, see Configuring multi-cluster Services.

Multi-cluster Services exports Services across clusters which allows cluster-to-cluster communication

Multi-cluster Gateways also use ServiceImports, but not for cluster-to-cluster load balancing. Instead, Gateways use ServiceImports as logical identifiers for a Service that exists in another cluster or that stretches across multiple clusters. The following HTTPRoute references a ServiceImport instead of a Service resource. By referencing a ServiceImport, this indicates that it is forwarding traffic to a group of backend Pods that run across one or more clusters.

kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: store-route
  namespace: store
  labels:
    gateway: multi-cluster-gateway
spec:
  hostnames:
  - "store.example.com"
  rules:
  - forwardTo:
    - backendRef:
        group: net.gke.io
        kind: ServiceImport
        name: store
      port: 8080

The following diagram shows how the HTTPRoute routes store.example.com traffic to store Pods on gke-1 and gke-2. The load balancer treats them as one pool of backends. If the Pods from one of the clusters becomes unhealthy, unreachable, or has no traffic capacity, then traffic load is balanced to the remaining Pods on the other cluster. New clusters can be added or removed with the store Service and ServiceExport. This will transparently add or remove backend Pods without any explicit routing configuration changes.

MCS resource

Exporting Services

At this point, the application is running across both clusters. Next, you will expose and export the applications by deploying Services and ServiceExports to each cluster.

  1. Save the following manifest to a file named store-west-service.yaml:

    apiVersion: v1
    kind: Service
    metadata:
      name: store
      namespace: store
    spec:
      selector:
        app: store
      ports:
      - port: 8080
        targetPort: 8080
    ---
    kind: ServiceExport
    apiVersion: net.gke.io/v1
    metadata:
      name: store
      namespace: store
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: store-west-1
      namespace: store
    spec:
      selector:
        app: store
      ports:
      - port: 8080
        targetPort: 8080
    ---
    kind: ServiceExport
    apiVersion: net.gke.io/v1
    metadata:
      name: store-west-1
      namespace: store
    
  2. Deploy this manifest to gke-west-1:

    kubectl apply -f store-west-service.yaml --context gke-west-1
    
  3. Save the following manifest to a file named store-east-service.yaml:

    apiVersion: v1
    kind: Service
    metadata:
      name: store
      namespace: store
    spec:
      selector:
        app: store
      ports:
      - port: 8080
        targetPort: 8080
    ---
    kind: ServiceExport
    apiVersion: net.gke.io/v1
    metadata:
      name: store
      namespace: store
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: store-east-1
      namespace: store
    spec:
      selector:
        app: store
      ports:
      - port: 8080
        targetPort: 8080
    ---
    kind: ServiceExport
    apiVersion: net.gke.io/v1
    metadata:
      name: store-east-1
      namespace: store
    
  4. Deploy this ServiceExport resource to gke-east-1:

    kubectl apply -f store-east-service.yaml --context gke-east-1 --namespace store
    
  5. Verify that the correct ServiceExports have been created in the cluster.

    kubectl get serviceexports --context CLUSTER_NAME --namespace store
    

    Replace CLUSTER_NAME with gke-west-1 and gke-east-1. The output should resemble the following:

    # gke-west-1
    NAME           AGE
    store          2m40s
    store-west-1   2m40s
    
    # gke-east-1
    NAME           AGE
    store          2m25s
    store-east-1   2m25s
    

    This demonstrates that the store Service contains store Pods across both clusters while the store-west-1 and store-east-1 Services only contain store Pods on their respective clusters. These overlapping Services are used to target the Pods across multiple clusters or a subset of Pods on a single cluster.

  6. After a few minutes verify that the accompanying ServiceImports have been automatically created by the multi-cluster Services controller across all clusters in the fleet.

    kubectl get serviceimports --context CLUSTER_NAME --namespace store
    

    Replace CLUSTER_NAME with gke-west-1 and gke-east-1. The output should resemble the following:

    # gke-west-1
    NAME           TYPE           IP                  AGE
    store          ClusterSetIP   ["10.112.31.15"]    6m54s
    store-east-1   ClusterSetIP   ["10.112.26.235"]   5m49s
    store-west-1   ClusterSetIP   ["10.112.16.112"]   6m54s
    
    # gke-east-1
    NAME           TYPE           IP                  AGE
    store          ClusterSetIP   ["10.72.28.226"]   5d10h
    store-east-1   ClusterSetIP   ["10.72.19.177"]   5d10h
    store-west-1   ClusterSetIP   ["10.72.28.68"]    4h32m
    

This demonstrates that all three Services are accessible from both clusters in the fleet. However, because there is only a single active config cluster per fleet, you can only deploy Gateways and HTTPRoutes that reference these ServiceImports in gke-west-1. When an HTTPRoute in the config cluster references these ServiceImports as backends, the Gateway can forward traffic to these Services no matter which cluster they are exported from.

Deploy the Gateway and HTTPRoute

Once the applications have been deployed, you can then configure a Gateway using the gke-l7-gxlb-mc GatewayClass. This Gateway creates an external HTTP(S) load balancer configured to distribute traffic across your target clusters.

  1. Save the following Gateway manifest to a file named external-http-gateway.yaml:

    kind: Gateway
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: external-http
      namespace: store
    spec:
      gatewayClassName: gke-l7-gxlb-mc
      listeners:
      - protocol: HTTP
        port: 80
        routes:
          kind: HTTPRoute
          selector:
            matchLabels:
              gateway: external-http
    

    This uses the gke-l7-gxlb-mc GatewayClass and targets HTTPRoutes with the label gateway: external-http.

  2. Apply the Gateway manifest to this example's gke-west-1 config cluster:

    kubectl apply -f external-http-gateway.yaml --context gke-west-1 --namespace store
    
  3. Save the following HTTPRoute manifest to a file named public-store-route.yaml:

    kind: HTTPRoute
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: public-store-route
      namespace: store
      labels:
        gateway: external-http
    spec:
      hostnames:
      - "store.example.com"
      rules:
      - forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store
          port: 8080
      - matches:
        - path:
            type: Prefix
            value: /west
        forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-1
          port: 8080
      - matches:
        - path:
            type: Prefix
            value: /east
        forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-east-1
          port: 8080
    

Once deployed, this HTTPRoute will configure the following routing behavior:

  • Requests to /west are routed to store Pods in the gke-west-1 cluster.
  • Requests to /east are routed to store Pods in thegke-east-1 cluster.
  • Requests to any other path are routed to either cluster, according to its health, capacity, and proximity to the requesting client.

The HTTPRoute enables routing to different subsets of clusters by using overlapping Services

Note that if all the Pods on a given cluster are unhealthy (or do not exist) then traffic to the store Service would only be sent to clusters that actually have store Pods. The existence of a ServiceExport and Service on a given cluster does not guarantee that traffic will be sent to that cluster. Pods must exist and be responding affirmatively to the load balancer health check or else the load balancer will just send traffic to healthy store Pods in other clusters.

  1. Apply the HTTPRoute manifest to this example's gke-1 config cluster:

    kubectl apply -f public-store-route.yaml --context gke-west-1 --namespace store
    

The following diagram shows the resources you've deployed across both clusters. Because gke-west-1 is the Gateway config cluster, it is the cluster in which our Gateway, HTTPRoutes, and ServiceImports are watched by the Gateway controller. Each cluster has a store ServiceImport and another ServiceImport specific to that cluster. Both point at the same Pods. This lets the HTTPRoute to specify exactly where traffic should go - to the store Pods on a specific cluster or to the store Pods across all clusters.

This is the Gateway and Multi-cluster Service resource model across both clusters

Note that this is a logical resource model, not a depiction of the traffic flow. The traffic path goes directly from the load balancer to backend Pods and has no direct relation to whichever cluster is the config cluster.

Validating deployment

You can now issue requests to our multi-cluster Gateway and distribute traffic across both GKE clusters.

  1. Validate that the Gateway and HTTPRoute have been deployed successfully by inspecting the Gateway status and events.

    kubectl describe gateway external-http --context gke-west-1 --namespace store
    

    Your output should look similar to the following:

    Spec:
      Gateway Class Name:  gke-l7-gxlb-mc
      Listeners:
        Port:      80
        Protocol:  HTTP
        Routes:
          Group:  networking.x-k8s.io
          Kind:   HTTPRoute
          Namespaces:
            From:  Same
          Selector:
            Match Labels:
              Gateway:  external-http
    Status:
      Addresses:
        Type:   IPAddress
        Value:  34.120.172.213
      Conditions:
        Last Transition Time:  1970-01-01T00:00:00Z
        Message:               Waiting for controller
        Reason:                NotReconciled
        Status:                False
        Type:                  Scheduled
    Events:
      Type     Reason  Age                     From                     Message
      ----     ------  ----                    ----                     -------
      Normal   UPDATE  29m (x2 over 29m)       global-gke-gateway-ctlr  store/external-http
      Normal   SYNC    59s (x9 over 29m)       global-gke-gateway-ctlr  SYNC on store/external-http was a success
    
  2. Once the Gateway has deployed successfully retrieve the external IP address from external-http Gateway.

    kubectl get gateway external-http -o=jsonpath="{.status.addresses[0].value}" --context gke-west-1 --namespace store
    

    Replace VIP in the following steps with the IP address you receive as output.

  3. Send traffic to the root path of the domain. This load balances traffic to the store ServiceImport which is across cluster gke-west-1 and gke-east-1. The load balancer sends your traffic to the closest region to you and you might not see responses from the other region.

    curl -H "host: store.example.com" http://VIP
    

    The output confirms that the request was served by Pod from the gke-west-1 cluster:

    {
    "cluster_name": "gke-east-1",
    "zone": "us-east1-b",
    "host_header": "store.example.com",
    "node_name": "gke-gke-east-1-default-pool-7aa30992-t2lp.c.agmsb-k8s.internal",
    "pod_name": "store-5f5b954888-dg22z",
    "pod_name_emoji": "⏭",
    "project_id": "agmsb-k8s",
    "timestamp": "2021-06-01T17:32:51"
    }
    
  4. Next send traffic to the /west path. This routes traffic to the store-west ServiceImport which only has Pods running on the gke-west-1 cluster. A cluster-specific ServiceImport like store-west enables an application owner to explicitly send traffic to a specific cluster, rather than letting the load balancer make the decision.

    curl -H "host: store.example.com" http://VIP/west
    

    The output confirms that the request was served by Pod from the gke-west-1 cluster:

    {
    "cluster_name": "gke-west-1", 
    "zone": "us-west1-a", 
    "host_header": "store.example.com",
    "node_name": "gke-gke-west-1-default-pool-65059399-2f41.c.agmsb-k8s.internal",
    "pod_name": "store-5f5b954888-d25m5",
    "pod_name_emoji": "🍾",
    "project_id": "agmsb-k8s",
    "timestamp": "2021-06-01T17:39:15",
    }
    
  5. Finally, send traffic to the /east path.

    curl -H "host: store.example.com" http://VIP/east
    

    The output confirms that the request was served by Pod from the gke-east-1 cluster:

    {
    "cluster_name": "gke-east-1",
    "zone": "us-east1-b",
    "host_header": "store.example.com",
    "node_name": "gke-gke-east-1-default-pool-7aa30992-7j7z.c.agmsb-k8s.internal",
    "pod_name": "store-5f5b954888-hz6mw",
    "pod_name_emoji": "🧜🏾",
    "project_id": "agmsb-k8s",
    "timestamp": "2021-06-01T17:40:48"
    }
    

Blue-green, multi-cluster routing via internal Gateway

The gke-l7-rilb GatewayClasses have many advanced traffic routing capabilities including traffic splitting, header matching, header manipulation, traffic mirroring, and more. In this example, you'll demonstrate how to use weight-based traffic splitting to explicitly control the traffic proportion across two GKE clusters.

This example goes through some realistic steps that a service owner would take in moving or expanding their application to a new GKE cluster. The goal of blue-green deployments is to reduce risk through multiple validation steps which confirm that the new cluster is operating correctly. This example walks through four stages of deployment:

  1. 100%-Header-based canary: Use HTTP header routing to send only test or synthetic traffic to the new cluster.
  2. 100%-Mirror traffic: Mirror user traffic to the canary cluster. This tests the capacity of the canary cluster by copying 100% of the user traffic to this cluster.
  3. 90%-10%: Canary a traffic split of 10% to slowly expose the new cluster to live traffic.
  4. 0%-100%: Cutover fully to the new cluster with the option of switching back if any errors are observed.

Blue-green traffic splitting across two GKE clusters

This example is similar to the previous one, except it deploys an internal multi-cluster Gateway instead. This deploys an internal HTTP(S) load balancer which is only privately accessible from within the VPC. You will use the clusters and same application that you deployed in the previous steps, except deploy them through a different Gateway.

Prerequisites

The following example builds on some of the steps in Deploying an external multi-cluster Gateway. Ensure that you have done the following steps before proceeding with this example:

  1. Enabling multi-cluster Gateways
  2. Deploying a demo application

This example uses the gke-west-1 and gke-west-2 clusters that you already set up. These clusters are in the same region because the gke-l7-rilb-mc GatewayClass is regional and only supports cluster backends in the same region.

  1. Deploy the Service and ServiceExports needed on each cluster. If you deployed Services and ServiceExports from the previous example then you already deployed some of these.

    kubectl apply --context gke-west-1 -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/gke-gateway-controller/multi-cluster-gateway/store-west-1-service.yaml
    kubectl apply --context gke-west-2 -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/gke-gateway-controller/multi-cluster-gateway/store-west-2-service.yaml
    

    It deploys a similar set of resources to each cluster:

    service/store created
    serviceexport.net.gke.io/store created
    service/store-west-2 created
    serviceexport.net.gke.io/store-west-2 created
    

Deploying the Gateway

The following Gateway is created from the gke-l7-rilb-mc GatewayClass. This is a regional Gateway which can only target GKE clusters in the same region. The Route selector gateway: multi-cluster-gateway-ilb is the same one that is applied to the HTTPRoute resource that is deployed next.

  1. If you have not done so already, create a proxy-only subnet for the internal load balancer. This is required for the internal load balancer to function correctly. It creates a subnet that is used to provide IP addressing to the load balancer proxies. Create the proxy-only subnet in the same region that your cluster is in.

  2. Save the following Gateway manifest to a file named internal-http-gateway.yaml:

    kind: Gateway
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: internal-http
      namespace: store
    spec:
      gatewayClassName: gke-l7-rilb-mc
      listeners:
      - protocol: HTTP
        port: 80
        routes:
          kind: HTTPRoute
          selector:
            matchLabels:
              gateway: internal-http
    
  3. Apply the Gateway manifest to this example's gke-west-1 config cluster:

    kubectl apply -f internal-http-gateway.yaml --context gke-west-1 --namespace store
    
  4. Validate that the Gateway has come up successfully. You can filter for just the events from this Gateway with the following command:

    kubectl get events --field-selector involvedObject.kind=Gateway,involvedObject.name=internal-http --context=gke-west-1 --namespace store
    

    The Gateway deployment was successful if the output resembles the following:

    LAST SEEN   TYPE     REASON   OBJECT                  MESSAGE
    22m         Normal   ADD      gateway/internal-http   store/internal-http
    6m50s       Normal   UPDATE   gateway/internal-http   store/internal-http
    11m         Normal   SYNC     gateway/internal-http   store/internal-http
    3m26s       Normal   SYNC     gateway/internal-http   SYNC on store/internal-http was a success
    

Header-based canary

Header-based canarying lets the service owner match synthetic test traffic that does not come from real users. This is an easy way of validating that the basic networking of the application is functioning without exposing users directly.

  1. Save the following manifest as a file named internal-route-stage-1.yaml.

    kind: HTTPRoute
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: internal-store-route
      namespace: store
      labels:
        gateway: internal-http
    spec:
      hostnames:
      - "store.example.internal"
      rules:
    # Matches for env=canary and sends it to store-west-2 ServiceImport
      - matches:
        - headers:
            values:
              env: canary
        forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-2
          port: 8080
    # All other traffic goes to store-west-1 ServiceImport
      - forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-1
          port: 8080
    

Once deployed, this HTTPRoute configures the following routing behavior:

  • Internal requests to store.example.internal without the env: canary HTTP header are routed to store Pods on the gke-west-1 cluster
  • Internal requests to store.example.internal with the env: canary HTTP header are routed to store Pods on the gke-west-2 cluster

The HTTPRoute enables routing to different clusters based on the HTTP headers

  1. Apply internal-route-stage-1.yaml to the gke-west-1 cluster:

    kubectl apply -f internal-route-stage-1.yaml --context gke-west-1 --namespace store
    

Validate that the HTTPRoute is functioning correctly by sending traffic to the Gateway IP address.

  1. Retrieve the internal IP address from internal-http.

    kubectl get gateway internal-http -o=jsonpath="{.status.addresses[0].value}" --context gke-west-1 --namespace store
    

    Replace VIP in the following steps with the IP address you receive as output.

  1. Send a request to the Gateway using the env: canary HTTP header. This will confirm that traffic is being routed to gke-west-2. Use a private client in the same VPC as the GKE clusters to confirm that requests are being routed correctly. The following command must be run on a machine that has private access to the Gateway IP address or else it will not function.

    curl -H "host: store.example.internal" -H "env: canary" http://VIP
    

    The output confirms that the request was served by a Pod from the gke-west-2 cluster:

    {
      "cluster_name": "gke-west-2", 
      "host_header": "store.example.internal",
      "node_name": "gke-gke-west-2-default-pool-4cde1f72-m82p.c.agmsb-k8s.internal",
      "pod_name": "store-5f5b954888-9kdb5",
      "pod_name_emoji": "😂",
      "project_id": "agmsb-k8s",
      "timestamp": "2021-05-31T01:21:55",
      "zone": "us-west1-a"
    }
    

Traffic mirror

This stage sends traffic to the intended cluster but also mirrors that traffic to the canary cluster.

  1. Save the following manifest as a file named internal-route-stage-2.yaml.

    kind: HTTPRoute
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: internal-store-route
      namespace: store
      labels:
        gateway: internal-http
    spec:
      hostnames:
      - "store.example.internal"
      rules:
    # Sends all traffic to store-west-1 ServiceImport
      - forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-1
          port: 8080
    # Also mirrors all traffic to store-west-2 ServiceImport
        filters:
        - type: RequestMirror
          requestMirror:
            backendRef:
              group: net.gke.io
              kind: ServiceImport
              name: store-west-2
            port: 8080
    
  2. Apply internal-route-stage-2.yaml to the gke-west-1 cluster:

    kubectl apply -f internal-route-stage-2.yaml --context gke-west-1 --namespace store
    
  3. Using your private client, send a request to the internal-http Gateway. Use the /mirror path so you can uniquely identify this request in the application logs in a later step.

    curl -H "host: store.example.internal" http://VIP/mirror
    
  4. The output confirms that the client received a response from a Pod in the gke-west-1 cluster:

    {
       "cluster_name": "gke-west-1", 
       "host_header": "store.example.internal",
       "node_name": "gke-gke-west-1-default-pool-65059399-ssfq.c.agmsb-k8s.internal",
       "pod_name": "store-5f5b954888-brg5w",
       "pod_name_emoji": "🎖",
       "project_id": "agmsb-k8s",
       "timestamp": "2021-05-31T01:24:51",
       "zone": "us-west1-a"
    }
    

    This confirms that the primary cluster is responding to traffic. You still need to confirm that the cluster you are migrating to is receiving mirrored traffic.

  5. Check the application logs of a store Pod on the gke-west-2 cluster. The logs should confirm that the Pod received mirrored traffic from the load balancer.

    kubectl logs deployment/store --context gke-west-2 -n store | grep /mirror
    
  6. This output confirms that Pods on the gke-west-2 cluster are also receiving the same requests, however their responses to these requests are not sent back to the client. The IP addresses seen in the logs are that of the load balancer's internal IP addresses which are communicating with your Pods.

    Found 2 pods, using pod/store-5f5b954888-gtldf
    2021-05-31 01:32:21,416 ... "GET /mirror HTTP/1.1" 200 -
    2021-05-31 01:32:23,323 ... "GET /mirror HTTP/1.1" 200 -
    2021-05-31 01:32:24,137 ... "GET /mirror HTTP/1.1" 200 -
    

Using mirroring is helpful to determine how traffic load will impact application performance without impacting responses to your clients in any way. It may not be necessary for all kinds of rollouts, but can be useful when rolling out large changes that could impact performance or load.

Traffic split

Traffic splitting is one of the most common methods of rolling out new code or deploying to new environments safely. The service owner sets an explicit percentage of traffic that is sent to the canary backends that is typically a very small amount of the overall traffic so that the success of the rollout can be determined with an acceptable amount of risk to real user requests.

  1. Save the following manifest as a file named internal-route-stage-3.yaml.

    kind: HTTPRoute
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: internal-store-route
      namespace: store
      labels:
        gateway: internal-http
    spec:
      hostnames:
      - "store.example.internal"
      rules:
    # 90% of traffic to store-west-1 ServiceImport
      - forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-1
          port: 8080
          weight: 90
    # 10% of traffic to store-west-2 ServiceImport
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-2
          port: 8080
          weight: 10
    
  2. Apply internal-route-stage-3.yaml to the gke-west-1 cluster:

    kubectl apply -f internal-route-stage-3.yaml --context gke-west-1 --namespace store
    
  3. Using your private client, send a continuous curl request to the internal- http Gateway.

    while true; do curl -H "host: store.example.internal" -s VIP | grep "cluster_name"; sleep 1; done
    

    The output will be similar to this, indicating that a 90/10 traffic split is occurring.

    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-2",
    "cluster_name": "gke-west-1",
    "cluster_name": "gke-west-1",
    ...
    

Doing a traffic split with a minority of the traffic enables the service owner to inspect the health of the application and the responses. If all the signals look healthy, then they may proceed to the full cutover.

Traffic cut over

The last stage of the blue-green migration is to fully cut over to the new cluster and remove the old cluster. If the service owner was actually onboarding a second cluster to an existing cluster then this last step would be different as the final step would have traffic going to both clusters. In that scenario a single store ServiceImport is recommended that has Pods from both gke-west-1 and gke-west-2 clusters. This allows the load balancer to make the decision of where traffic should go for an active-active application, based on proximity, health, and capacity.

  1. Save the following manifest as a file named internal-route-stage-4.yaml.

    kind: HTTPRoute
    apiVersion: networking.x-k8s.io/v1alpha1
    metadata:
      name: internal-store-route
      namespace: store
      labels:
        gateway: internal-http
    spec:
      hostnames:
      - "store.example.internal"
      rules:
    # No traffic to the store-west-1 ServiceImport
      - forwardTo:
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-1
          port: 8080
          weight: 0
    # All traffic to the store-west-2 ServiceImport
        - backendRef:
            group: net.gke.io
            kind: ServiceImport
            name: store-west-2
          port: 8080
          weight: 100
    
  2. Apply internal-route-stage-4.yaml to the gke-west-1 cluster:

    kubectl apply -f internal-route-stage-4.yaml --context gke-west-1 --namespace store
    
  3. Using your private client, send a continuous curl request to the internal- http Gateway.

    while true; do curl -H "host: store.example.internal" -s VIP | grep "cluster_name"; sleep 1; done
    

    The output will be similar to this, indicating that all traffic is now going to gke-west-2.

    "cluster_name": "gke-west-2",
    "cluster_name": "gke-west-2",
    "cluster_name": "gke-west-2",
    "cluster_name": "gke-west-2",
    ...
    

This final step completes a full blue-green application migration from one GKE cluster to another GKE cluster.

What's next