Migrate to multi-cluster networking with Google Kubernetes Engine (GKE) Autopilot and Standard clusters

Last reviewed 2022-02-17 UTC

Autopilot Standard

---

Multi-cluster networking is a valuable tool that enables use cases like regional high availability, globally distributed proximity to users for lower latency, and organizational isolation between teams. Google Kubernetes Engine (GKE) provides built-in capabilities for multi-cluster networking that you can enable and use at scale across a fleet of GKE clusters. This feature also lets you combine, or migrate deployed infrastructure between, GKE Standard and Autopilot to meet the architectural needs of each application.

This document demonstrates these features through several deployment topologies. You learn how to take an application deployed in a single GKE cluster and migrate it to a multi-cluster deployment across GKE Standard and Autopilot clusters. You use GKE multi-cluster Services for east-west traffic and multi-cluster Gateways to enable multi-cluster north-south networking.

The document is for cloud architects and operations teams that use, or plan to use, GKE to deploy services across multiple Kubernetes clusters. This document assumes that you're familiar with Kubernetes.

GKE Standard and GKE Autopilot

GKE provides a managed Kubernetes deployment with a full feature set including a high availability control plane. GKE clusters can be started quickly, and scale to up to 15,000 nodes. With GKE Autopilot clusters, Google manages the infrastructure, including the control plane and the nodes. If you want to configure and manage your nodes instead, GKE provides Standard mode.

For more information on the differences between modes, see Choose a cluster mode of operation.

Multi-cluster Services and multi-cluster Gateways

Kubernetes can run using a single control plane across disparate cloud zones to provide resiliency and higher availability for your services. GKE takes this one step further and provides GKE multi-cluster Services (MCS) that provides a cross-cluster service discovery and invocation mechanism. Services that use this feature are discoverable and accessible across clusters with a virtual IP, which matches the behavior of a ClusterIP Service accessible in a cluster. This approach allows for the following benefits:

• Services can be load-balanced across multiple clusters in the same region or different regions (east-west traffic).
• Cross-region service high availability options are achievable.
• Stateful and stateless workloads can be deployed and managed across separate clusters.
• Shared services are available across clusters.

To learn more about how to deploy MCS, see Configuring multi-cluster Services.

GKE provides an implementation of the Kubernetes Gateway API that uses the GKE Gateway controller. Gateway allows GKE to deploy Google Cloud load balancers to provide inbound (north-south) traffic routing for services deployed on GKE. GKE also provides multi-cluster Gateways (MCG) which extend the GKE Gateway controller to provision load balancers that route traffic to services deployed on disparate GKE clusters.

The following diagram shows how, when you combine MCS and MCG, you are able to manage the complementary aspects of service deployment and traffic routing from a single control plane:

For more information, see Deploying multi-cluster Gateways.

Note: In a multi-cluster scenario where clusters belong to different custom subnets and are registered to the same fleet, you must create a firewall rule to allow cross-cluster traffic. For more information, see create firewall rule.

Overview of migration

GKE multi-cluster networking capabilities benefit workloads of various profiles. For example, you might have stateless components with bursty traffic that you wish to move to Autopilot because of its more efficient cost model.

Or, you might want to place your application frontends closer to users. This approach provides lower latency and caching that improves application performance and user experience. At the same time, you might have some stateful components that your application relies on that can only reside in one location. This configuration requires north-south, multi-cluster load balancing to send client traffic to the correct cluster in that location. You also need east-west, multi-cluster load balancing to send traffic between clusters to reach the stateful components.

This document uses the Online Boutique cloud microservices demo application to demonstrate a multi-cluster pattern that can be used to enhance the deployment of the single zone demo. You start with a single-zone version of the application. You then add elements of high availability and resilience by using multi-cluster Services and multi-cluster Gateways, and reduce operational toil by taking advantage of Autopilot.

Initial single-cluster deployment

In the following diagram, the Online Boutique application is initially deployed to a single GKE Standard mode cluster named std-west, and is exposed by using a LoadBalancer Service:

Migrate to multi-cluster Services

In the next intermediate step, you create two additional clusters, and the stateless services are deployed in additional regions. You create two GKE Autopilot clusters named auto-east and auto-central in two separate regions distinct from the single std-west GKE Standard cluster, and register the clusters to the Google Cloud fleet.

Fleets are a Google Cloud concept for logically organizing clusters and other resources, and let you use and manage multi-cluster capabilities and apply consistent policies across your systems.

You export the cartservice on the std-west cluster in the onlineboutique namespace to the new fleet clusters by using ServiceExport. You deploy Online Boutique's frontend Service on all three clusters and expose it through a ClusterIP service. You then export the service to the fleet using ServiceExports. Services such as the middleware layer of the Online Boutique (like productcatalog, shipping, and adservice) are also deployed to all three clusters.

A Pod that runs in any cluster in the fleet can access an exported Service by sending a request to the ClusterSet URI for that service. The request routes to an endpoint that backs the service.

The frontend Service is able to consume the middleware services (such as productcatalogservice or currencyservice) locally in the same cluster. This architecture helps keep incoming requests local to the regions whose frontend responds to the request, and avoids unnecessary inter-region network traffic charges.

The following diagram illustrates the two multi-cluster Services. The stateless frontend Service is deployed to three clusters, and the stateful backend cartservice is deployed to one cluster. The diagram also shows that in this intermediate step, inbound traffic for the frontend service remains routed to the original GKE Standard cluster in us-west1 using an external passthrough Network Load Balancer created by the frontend-external LoadBalancer Service:

Migrate to multi-cluster Gateway

In the final step, you route inbound traffic for the frontend Service from external client requests to services in multiple clusters in the fleet using a multi-cluster Gateway.

A fourth cluster named config-central is added to the fleet to host and manage the configuration for the Gateway and HTTPRoute resources that are created as part of this configuration. The HTTPRoute resource maps the / prefix to the frontend ServiceImport. Traffic for the Online Boutique's frontend is sent to a healthy endpoint in one of the available regions. This approach adds elements of high availability to the Online Boutique application architecture.

In the following diagram, the multi-cluster Gateway deploys a Global Cloud Load Balancer that routes external traffic to the stateless frontend Service deployed on each of the three application clusters in the fleet.

In the final state, this opinionated pattern demonstrates loose coupling between the stateful (cartservice and redis-cart) and stateless portions of the application (frontend, emailservice, checkoutservice, recommendationservice, paymentservice, productcatalogservice, currencyservice, shippingservice, and adservice). While outside the scope of this document, this approach gives you a future opportunity to add resilience and high availability to the stateful services layer.

Objectives

• Create and configure GKE Standard and Autopilot clusters.
• Deploy Online Boutique to a zonal GKE Standard cluster.
• Export multi-cluster Services.
• Deploy manifests to Standard and Autopilot clusters.
• Enable and configure multi-cluster Gateways.
• Test multi-region application behavior.

Costs

In this document, you use the following billable components of Google Cloud:

• GKE
• Compute Engine for GKE worker nodes
• Cloud Load Balancing
• Multi Cluster Ingress

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

Before you begin, ensure you have the following requirements met:

• It is recommended that you use a new project for this guide, as the easiest way to clean up is to delete the project once you are finished.
• This guide assumes you have the Owner IAM role for your Google Cloud project. For production or real-world settings, it is best practice to scope permissions to least privilege. For more information, see Using IAM securely and Manage identity and access.
• Familiarize yourself with the Online Boutique microservices demo application architecture.

Prepare the environment

In this guide, you use Cloud Shell to enter commands. Cloud Shell gives you access to the command line in the Google Cloud console and includes Google Cloud SDK and other tools, such as the Google Cloud CLI. Cloud Shell appears as a window at the bottom of the Google Cloud console. It can take several minutes to initialize, but the window appears immediately.

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

2. In Cloud Shell, define environment variables used in this guide. Replace PROJECT_ID with your own project ID:

   export PROJECT=PROJECT_ID
   gcloud config set project ${PROJECT}
   

3. Enable the services required for this document:

   gcloud services enable \
       gkehub.googleapis.com \
       multiclusteringress.googleapis.com \
       dns.googleapis.com \
       trafficdirector.googleapis.com \
       cloudresourcemanager.googleapis.com \
       multiclusterservicediscovery.googleapis.com \
       container.googleapis.com
   
   gcloud container fleet multi-cluster-services enable
   

   Multi-cluster Services manages Google Cloud components like Cloud DNS, firewall rules, and Traffic Director, so these APIs must also be enabled. For more information, see Traffic Director overview.

   The output is similar to the following example:

   Operation "operations/acf.p2-822685001869-ee4ebe78-6dd8-465e-b0fd-3b0e5f964bad"
   finished successfully.
   
   Waiting for Feature Multi-cluster Services to be created...done.
   

4. Verify that multi-cluster Services shows the ACTIVE state:

   gcloud container fleet multi-cluster-services describe
   

   The output is similar to the following example:

   createTime: '2021-11-30T21:59:25.245190894Z'
   name: projects/PROJECT_ID/locations/global/features/multiclusterservicediscovery
   resourceState:
     state: ACTIVE
   spec: {}
   updateTime: '2021-11-30T21:59:27.459063070Z'
   

   If the value of state is not ACTIVE, see the troubleshooting section of the MCS documentation.

Create and configure GKE clusters

To demonstrate the multi-cluster pattern in this guide, you use three application clusters in three separate cloud regions, and one cluster to host the configuration for Gateway resources. You register all clusters with the fleet associated with your project. A Google Cloud project can only have a single fleet associated with it. This project is known as the fleet host project.

1. Create Standard and Autopilot GKE clusters:

   gcloud container clusters create std-west \
       --zone us-west1-a \
       --num-nodes=6 \
       --enable-ip-alias \
       --release-channel regular \
       --workload-pool=${PROJECT}.svc.id.goog \
       --async
   
   gcloud container clusters create-auto auto-east \
       --region us-east1 \
       --release-channel regular \
       --async
   
   gcloud container clusters create-auto auto-central \
       --region us-central1 \
       --release-channel regular \
       --async
   
   gcloud container clusters create config-central \
       --region us-central1 \
       --num-nodes=1 \
       --enable-ip-alias \
       --release-channel regular \
       --workload-pool=${PROJECT}.svc.id.goog \
       --async
   

   Workload Identity Federation for GKE is enabled by default on GKE Autopilot clusters so you don't have to use the --workload-pool flag when you create those clusters like you do with the GKE Standard clusters.

2. Wait for the clusters' STATUS to change from PROVISIONING to RUNNING. This process can take up to 10 minutes. You can monitor the progress by using a watch loop while you grab a cup of caffeine or do some light stretching exercises before you proceed with the rest of the document:

   watch -n 20 --difference=permanent "gcloud container clusters list"
   

   The output is similar to the following example:

   NAME: auto-central
   LOCATION: us-central1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 107.178.213.138
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 3
   STATUS: PROVISIONING
   
   NAME: config-central
   LOCATION: us-central1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP:
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 9
   STATUS: PROVISIONING
   
   NAME: auto-east
   LOCATION: us-east1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 35.229.88.209
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 3
   STATUS: PROVISIONING
   
   NAME: std-west
   LOCATION: us-west1-a
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 35.197.93.113
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 6
   STATUS: PROVISIONING
   

3. After all the clusters are in the RUNNING state, press CTRL-C to interrupt the command.

4. Add an Identity and Access Management (IAM) policy binding granting the fleet host project MCS service account the Network User role for its own project:

   gcloud projects add-iam-policy-binding ${PROJECT} \
       --member "serviceAccount:${PROJECT}.svc.id.goog[gke-mcs/gke-mcs-importer]" \
       --role "roles/compute.networkViewer"
   

   You use Workload Identity Federation for GKE to grant the MCS service read access to your project VPC network configuration. As a result, the fleet host project's MCS Importer GKE service account needs this role.

   The output is similar to the following example:

   - members:
     - serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]
     role: roles/compute.networkViewer
   [...]
   

5. Register the GKE Standard and Autopilot clusters to your project's fleet. See Registering a cluster for more details. This step can take up to 5 minutes:

   gcloud container fleet memberships register std-west \
       --gke-cluster us-west1-a/std-west \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register auto-east \
       --gke-cluster us-east1/auto-east \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register auto-central \
       --gke-cluster us-central1/auto-central \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register config-central \
       --gke-cluster us-central1/config-central \
       --enable-workload-identity \
       --project=${PROJECT}
   

   For each command, the output is similar to the following example:

   Waiting for membership to be created...done.
   Created a new membership [projects/PROJECT_ID/locations/global/memberships/std-west] for the cluster [std-west]
   Generating the Connect Agent manifest...
   Deploying the Connect Agent on cluster [std-west] in namespace [gke-connect]...
   Deployed the Connect Agent on cluster [std-west] in namespace [gke-connect].
   Finished registering the cluster [std-west] with the Hub.
   

6. Connect to the clusters and generate kubeconfig entries:

   gcloud container clusters get-credentials std-west \
       --zone us-west1-a --project $PROJECT
   
   gcloud container clusters get-credentials auto-east \
       --region us-east1 --project $PROJECT
   
   gcloud container clusters get-credentials auto-central \
       --region us-central1 --project $PROJECT
   
   gcloud container clusters get-credentials config-central \
       --region us-central1 --project $PROJECT
   

   For each command, the output is similar to the following example:

   Fetching cluster endpoint and auth data.
   kubeconfig entry generated for std-west.
   

7. Rename contexts for clusters so that they are easier to work with in the rest of this document:

   kubectl config rename-context \
       gke_${PROJECT}_us-west1-a_std-west \
       std-west
   
   kubectl config rename-context \
       gke_${PROJECT}_us-east1_auto-east \
       auto-east
   
   kubectl config rename-context \
       gke_${PROJECT}_us-central1_auto-central \
       auto-central
   
   kubectl config rename-context \
       gke_${PROJECT}_us-central1_config-central \
       config-central
   

   In this guide, the contexts are named based on their location. Although you can provide alternate names, the remaining steps in this guide use the names used in this step.

Deploy Online Boutique on GKE Standard

In the first step of the demonstration deployment, you deploy the full set of Online Boutique application services to the single GKE Standard cluster std-west in us-west1.

1. Create the namespace onlineboutique on std-west:

   kubectl create namespace onlineboutique --context std-west
   

   The output is similar to the following example:

   namespace/onlineboutique created
   

2. Clone the Online Boutique GitHub repository and set up a WORKDIR variable:

   cd ~
   
   git clone --branch release/v0.4.1 \
       https://github.com/GoogleCloudPlatform/microservices-demo.git
   
   cd microservices-demo/release && export WORKDIR=`pwd`
   

3. Deploy Online Boutique on std-west. This process creates Deployments and Services for all of Online Boutique's microservices, and includes a LoadBalancer type Service which externally exposes Online Boutique's frontend service:

   cd $WORKDIR
   
   kubectl apply -f kubernetes-manifests.yaml \
       -n onlineboutique --context=std-west
   

4. Wait for the LoadBalancer Service to get an external IP:

   watch -n 20 --difference=permanent \
        "kubectl get svc frontend-external -n onlineboutique --context=std-west"
   

   Initially, the output is similar to the following example:

   NAME                TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
   frontend-external   LoadBalancer   10.60.5.62   <pending>     80:30359/TCP   43s
   

   When the Service is ready the EXTERNAL-IP column displays the public IP address of the load balancer.

5. After the Service is ready, get the external IP address of the load balancer and use curl to verify that the frontend is ready. If this curl command returns an error, wait a few moments before you try again:

     curl $(kubectl get svc frontend-external \
         -n onlineboutique --context=std-west \
         -o=jsonpath="{.status.loadBalancer.ingress[0].ip}") | \
           grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-b7bddcc97-wdjsk
   

You now have a single-zone version of Online Boutique running in us-west1-a. You can also use a web browser to navigate to the external IP assigned to the frontend-external LoadBalancer Service to access the application and observe its behavior. This initial single deployment is shown in the following diagram:

Export cartservice as a multi-cluster Service

In this section, you start to add elements of high availability to the application. You export the backend cartservice as a multi-cluster Service to the GKE Autopilot clusters.

1. Create the namespace onlineboutique on the remaining clusters:

   kubectl create namespace onlineboutique --context auto-east
   
   kubectl create namespace onlineboutique --context auto-central
   
   kubectl create namespace onlineboutique --context config-central
   

   For each command, the output is similar to the following example:

   namespace/onlineboutique created
   

2. Export cartservice from the std-west cluster to all other clusters in the ClusterSet. The ServiceExport object registers the cartservice Service, with GKE multi-cluster Services, for export to all clusters in the fleet that have the namespace onlineboutique present. For more details, see registering a service for export.

   cat <<EOF>> $WORKDIR/cartservice-export.yaml
   kind: ServiceExport
   apiVersion: net.gke.io/v1
   metadata:
    namespace: onlineboutique
    name: cartservice
   
   EOF
   
   kubectl apply -f $WORKDIR/cartservice-export.yaml \
       -n onlineboutique --context=std-west
   

Apply application manifests for multi-cluster pattern

In this section you apply two curated manifests to deploy the multi-cluster pattern. These manifests contain selected portions of the kubernetes-manifests.yaml that you previously applied to the std-west cluster:

• The first manifest is used for the frontend Deployment, Service, and ServiceExport.
• The second manifest is used to deploy the middleware Services (emailservice, checkoutservice, recommendationservice, paymentservice, productcatalogservice, currencyservice, shippingservice, and adservice) to all regions that have a frontend running. By keeping a request local to a region for as long as possible, you avoid unnecessary inter-region network traffic charges.

A Pod that runs in any cluster in the fleet can access an exported Service by sending a request to the ClusterSet URI for that service in the format SERVICE_NAME.NAMESPACE.svc.clusterset.local. For example, the frontend Deployments in all three example clusters are able to consume the cartservice, in the onlineboutique namespace, by making a request to cartservice.onlineboutique.svc.clusterset.local.

For this reason, in each manifest, the hostname for the cartservice has been updated to its ClusterSet URI. This step is critical. If this service hostname is not updated, the frontend Service would ask kube-dns for cartservice instead of cartservice.onlineboutique.svc.clusterset.local. This behavior would result in HTTP Status 500 errors on clusters where a local version of cartservice is not available, and cause the frontend pods to be unhealthy.

1. Set an environment varianble for the GitHub repository that contains the manifests:

   export MANIFEST_REPO_PATH=https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/docs/cluster-migration
   

2. Apply the manifests to deploy the frontend layer to all three workload clusters:

   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=std-west
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=auto-east
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=auto-central
   

3. Apply the manifests to deploy the middleware layer to all three workload clusters:

   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=std-west
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=auto-east
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=auto-central
   

You now have the frontend Deployment, Service, and ServiceExport active in clusters std-west, auto-east, and auto-central. You also have locally running Online Boutique middleware services in each cluster. However, external traffic is still routed only to the Service that runs in the initial cluster in us-west1, as shown in the following diagram:

Enable and configure multi-cluster Gateways

In this section, you route traffic to and load balance external traffic across frontends in all three clusters. To achieve this configuration, you use multi-cluster Gateways (MCG). This document follows the directions to set up MCG as described in Enabling multi-cluster Gateways.

In this guide you use the config-central cluster to host the configuration for Gateway resources.

1. Confirm that all clusters have successfully registered to the fleet:

   gcloud container fleet memberships list --project=$PROJECT
   

   The following example output shows that all the clusters are successfully registered:

   NAME: auto-central
   EXTERNAL_ID: 21537493-32ea-4a41-990d-02be2c1b319f
   
   NAME: config-central
   EXTERNAL_ID: 4369423e-ea7b-482d-a0eb-93b560e67b98
   
   NAME: std-west
   EXTERNAL_ID: 7fcb048b-c796-476b-9698-001a00f91ab3
   
   NAME: auto-east
   EXTERNAL_ID: aae2d2ff-b861-4a38-bcaf-612f14810012
   

2. Install the Gateway API custom resource definition on the config-central cluster:

   kubectl --context=config-central kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.5.0" \
       | kubectl apply -f -
   

   This step installs the Gateway API custom resource definitions including the GatewayClass, Gateway, and HTTPRoute resources. The custom resource definitions are maintained by the Kubernetes Network Special Interest Group. Once installed, you can use the GKE Gateway controller.

3. Enable Multi Cluster Ingress for your fleet if you haven't done so already. Enabling this feature also enables the multi-cluster Gateway controller.

   gcloud container fleet ingress enable \
       --config-membership=config-central \
       --project=$PROJECT
   
   gcloud container fleet ingress describe --project=$PROJECT
   

   The output is similar to the following example:

   createTime: '2021-12-08T23:10:52.505888854Z'
   name: projects/PROJECT_ID/locations/global/features/multiclusteringress
   resourceState:
     state: ACTIVE
   spec:
     multiclusteringress:
       configMembership: projects/zl-mcs-expf61cbd13/locations/global/memberships/config-central
   state:
     state:
       code: OK
       description: Ready to use
       updateTime: '2021-12-08T23:11:37.994971649Z'
   updateTime: '2021-12-08T23:11:38.098244178Z'
   

   If the value of state is not ACTIVE, see Troubleshooting and operations for Multi Cluster Ingress.

4. Confirm that GatewayClasses are available on the config-central cluster:

   kubectl get gatewayclasses --context=config-central
   

   The output is similar to the following example:

   NAME                                  CONTROLLER                  AGE
   gke-l7-global-external-managed        networking.gke.io/gateway   18s
   gke-l7-global-external-managed-mc     networking.gke.io/gateway   19s
   gke-l7-regional-external-managed      networking.gke.io/gateway   18s
   gke-l7-regional-external-managed-mc   networking.gke.io/gateway   19s
   gke-l7-gxlb                           networking.gke.io/gateway   74s
   gke-l7-gxlb-mc                        networking.gke.io/gateway   16s
   gke-l7-rilb                           networking.gke.io/gateway   74s
   gke-l7-rilb-mc                        networking.gke.io/gateway   16s
   

   Different GatewayClass resources have different capabilities. For more information on when to use which type, see GatewayClass capabilities.

5. Deploy the external-http Gateway resource to config-central:

   cat <<EOF>> $WORKDIR/external-http-gateway.yaml
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: external-http
     namespace: onlineboutique
   spec:
     gatewayClassName: gke-l7-global-external-managed-mc
     listeners:
     - protocol: HTTP
       port: 80
       name: http
   EOF
   
   kubectl apply -f external-http-gateway.yaml \
       -n onlineboutique --context=config-central
   

   As indicated by the gatewayClassName field, this resource is of GatewayClass gke-l7-global-external-managed-mc which manages Layer 7 external Cloud Load Balancing and exposes the multi-cluster application

6. Deploy the HTTPRoute named public-frontend-route on config-central:

   cat <<EOF>> $WORKDIR/public-frontend-route.yaml
   kind: HTTPRoute
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: public-frontend-route
     namespace: onlineboutique
   spec:
     parentRefs:
     - name: "external-http"
     hostnames:
     - "store.example.com"
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /
       backendRefs:
       - name: frontend
         group: net.gke.io
         kind: ServiceImport
         port: 80
   EOF
   
   kubectl apply -f public-frontend-route.yaml \
       -n onlineboutique --context=config-central
   

   When you deploy the HTTPRoute resource it creates an external Layer 7 Cloud Load Balancing resource and exposes the frontend ServiceImport backed by the frontend services that run in the std-west, auto-east, and auto-central clusters.

   Note: Alternatively, you can replace store.example.com in this manifest with a hostname from a domain you or your team owns, and point the respective DNS record to the external IP address of the load balancer created by this step. Requests destined for that hostname would then be directed to this load balancer.

   The following diagram shows how, after the multi-cluster Gateway is deployed, traffic can be routed to any of the frontend multi-cluster Services on any of the three application clusters:

   

7. Wait for the load balancer to be ready with a provisioned external IP address before proceeding to the next step. It can take up to 10 minutes for the IP address to be assigned. You can monitor progress by using a watch loop. The load balancer has a name in the pattern like gkemcg-onlineboutique-external-http-k09mfhk74gop:

   watch -n 20 --difference=permanent \
       "gcloud compute forwarding-rules list \
           | grep -A 5 NAME..*external-http"
   

   The output is similar to the following example:

   NAME: gkemcg-onlineboutique-external-http-k09mfhk74gop
   REGION:
   IP_ADDRESS: 34.149.29.176
   IP_PROTOCOL: TCP
   TARGET: gkemcg-onlineboutique-external-http-k09mfhk74gop
   

8. Once the load balancer is ready, run the following command in Cloud Shell to export the external IP address of the load balancer created through the application of external-http-gateway.yaml and public-frontend-route.yaml manifests:

   export EXTERNAL_LB_IP=$(kubectl --context=config-central \
                               -n onlineboutique get gateway external-http \
                               -o=jsonpath='{.status.addresses[0].value}')
   

9. When you send a request to the load balancer with the appropriate headers, it returns the HTML contents served by the frontend service. For example, since you configured the HTTPRoute resource to map the store.example.com hostname to the frontend ServiceImport, you must supply the HOST header when you make the HTTP request. If the following curl example returns an error, wait a few minutes and try again:

   curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
       grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-f<br/>
   <b>Pod: </b>frontend-7c7d596ddc-jdh8f
   

   Note: If the matching headers, as configured in the HTTPRoute, are not supplied when you make the request, the client receives an HTTP status 404 error. Any of the three frontend pods that run in the serving path can respond to the HTTP request inbound to the global load balancer. The global load balancer can route requests to healthy backends located closest to where the request originated.

Test application multi-region routing behavior

One of the powerful features obtained by using multi-cluster Services and multi-cluster Gateways is that external requests are routed to the geographically closest cluster.

To test application multi-region behavior, generate traffic that originates from the various regions where you have clusters deployed. Create three small pods, one in each of the serving clusters (std-west, auto-east, and auto-central), that you can use to send HTTP requests to the load balancer endpoint. The results let you see which frontend Pod responds.

1. Create the client pods:

   kubectl run --context=std-west \
               --image=radial/busyboxplus:curl client-west \
               -- sh -c 'while sleep 3600; do :; done'
   
   kubectl run --context=auto-east \
               --image=radial/busyboxplus:curl client-east \
               -- sh -c 'while sleep 3600; do :; done'
   
   kubectl run --context=auto-central \
               --image=radial/busyboxplus:curl client-central \
               -- sh -c 'while sleep 3600; do :; done'
   

2. After the pods are running, use a curl command to send a request to the load balancer endpoint from the client Pod in the std-west cluster and review the response:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-7cf48b79cf-trzc4
   

3. Run the same curl request from the client Pod in the auto-east cluster, and look at the response:

   kubectl exec -it --context=auto-east client-east \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-east<br/>
   <b>Zone: </b>us-east1-d<br/>
   <b>Pod: </b>frontend-6784b6df98-scdws
   

   As this is an Autopilot cluster, the cluster may need to provision additional resources to schedule the Pod. If you see output similar to the following example, wait a moment and try again:

    Error from server (BadRequest): pod client-east does not have a host assigned
   

4. Run the curl from the client Pod in the auto-central cluster and check the response:

   kubectl exec -it --context=auto-central client-central \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
           grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-b<br/>
   <b>Pod: </b>frontend-6784b6df98-x2fv4
   

   These results confirm that traffic routes to the corresponding pods in the locations closest to request origin.

Test application multi-region resiliency

In addition to efficient traffic routing, running your services in multiple regions provides resiliency in the rare, yet still possible, case of infrastructure failure.

Test the behavior by deleting the frontend Deployments in specific clusters and then retry the curl command from the client Pod in those regions. Observe that the application is still available, and look at the location of the Pod that responds to the request.

1. Run the curl command from the client-west Pod in the std-west cluster, and see that the result comes from the frontend in us-west1:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP  | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-7cf48b79cf-trzc4
   

2. Delete the frontend Deployment in the std-west cluster:

   kubectl delete deploy frontend \
       -n onlineboutique --context=std-west
   

   The output is similar to the following example:

   deployment.apps "frontend" deleted
   

3. Send another request from the client-west Pod in the std-west cluster. You should see a response from one of the remaining frontend Deployments located in the auto-east or auto-central clusters:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   Output similar to the following example indicates the location of the healthy Pod that responds to this request:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-b<br/>
   <b>Pod: </b>frontend-6784b6df98-x2fv4
   

   or

   <b>Cluster: </b>auto-east<br/>
   <b>Zone: </b>us-east1-d<br/>
   <b>Pod: </b>frontend-6784b6df98-scdws
   

   Run the command several times to see alternating results.

With this demonstration deployment, you have added elements of resiliency and geographic distribution to the Online Boutique application using multi-cluster Services and multi-cluster Gateways. Requests are routed to the closest geographical region and even if the frontend or middleware services in a region experience issues, the end user is still able to successfully use the application.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Read more about Google Cloud fleet and multi-cluster management.
• Learn about how to register a cluster, including GKE Enterprise clusters, to a Google Cloud fleet.
• Dive deeper into the Kubernetes Gateway API and the GKE Gateway.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.