Deploying Gateways

This page describes how to deploy Kubernetes Gateway resources for load balancing ingress traffic to a single Google Kubernetes Engine (GKE) cluster.

For deploying Gateways to load balance ingress traffic across multiple clusters (or fleet), see Deploying Multi-Cluster Gateways.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

GKE Gateway controller requirements

  • For Standard, GKE version 1.24 or later.
  • For Autopilot, GKE version 1.26 or later.
  • Google Cloud CLI version 407.0.0 or later.
  • The Gateway API is supported on VPC-native clusters only.
  • If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
  • Your cluster must have the HttpLoadBalancing add-on enabled.
  • If you are using Istio, you must upgrade Istio to one of the following versions:
    • 1.15.2 or later
    • 1.14.5 or later
    • 1.13.9 or later.
  • If you are using Shared VPC, then in the host project, you need to assign the Compute Network User role to the GKE Service account for the service project.

Restrictions and limitations

When using GKE Gateway, understand the following limitations and restrictions:

  • GKE GatewayClasses support different capabilities depending on the load balancer they use. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.

    Best practice:

    For optimal performance, limit the number of Gateways to a maximum of 100. Exceeding this limit can affect performance or result in increased latency.

  • You cannot use a FrontendConfig or a BackendConfig to configure a Gateway. You must use a Policy.

  • GKE Gateway behaves differently than Ingress, in that Gateway does not infer health check parameters. If your Service does not return 200 for requests to GET /, or you have other tuned pod readiness checks, you need to configure a HealthCheckPolicy for your service.

  • You cannot specify a port number directly in the hostname (for example, web.example.com:80) for traffic routing.

  • You can view the load balancer resources that GKE creates for Gateways in the Google Cloud console, but these resources don't reference the Gateway or GKE cluster they are attached to.

  • You can't automatically generate a Google-managed SSL certificate with Gateways but you can manually create and reference a Google-managed SSL certificate. For more information, see Secure a Gateway.

  • HTTPRoute is the only Route type supported. TCPRoutes, UDPRoutes, and TLSRoutes are not supported. To see a list of fields the GKE Gateway controller supports, see GatewayClass capabilities.

  • Custom request and response headers with Gateway or path redirects and URL rewrites with Gateway is only available on GKE version 1.27 or later.

  • For custom request and response headers with Gateway and path redirects and URL rewrites with Gateway, the GatewayClass gke-l7-gxlb is not supported.

  • When configuring HTTPRoute custom request and response headers, the following Google Cloud variables are not supported:

    • cdn_cache_id (Cloud CDN is not supported with GKE Gateway)
    • cdn_cache_status (Cloud CDN is not supported with GKE Gateway)
    • origin_request_header (CORS policies are not supported with GKE Gateway)

  • GKE Gateway does not support the Cloud CDN load balancing feature.

  • Mutual TLS custom headers are not supported (mTLS with GKE Gateway is not supported)

  • Google Cloud classic Application Load Balancer limitations apply to the GKE Gateway. In addition, you can't configure a custom Host response header in the backend service.

  • Path redirects and URL rewrites are mutually exclusive, you can't use both filters at the same time in the same rules.

  • Redirecting traffic to a different port is not supported with Cloud Load Balancing. To see the list of fields the GKE Gateway controller supports, see GatewayClass capabilities.

  • GKE Gateway does not support Wildcards, regular expressions, and dynamic URLs.

  • If you specify a Gateway with a regional external gateway class, the controller provisions an internal IP address instead of the external address. To learn how to use a named address with the regional external Application Load Balancer, see deploy a regional external Gateway.

  • Gateway utilizes Standalone NEGs for provisioning Network Endpoint Groups. To ensure that the Gateway controller properly reconciles the load balancer configuration, you cannot modify the cloud.google.com/neg annotation for a Service that is part of the Gateway.

  • GKE Gateway does not support referencing a Service that is also referenced by a GKE Ingress.

  • When a Gateway is configured to provision an IP address, changing the Gateway.spec.gatewayClass is not supported. To ensure that the Gateway controller properly reconciles the load balancer, delete the existing Gateway and re-deploy the manifest with the updated gatewayClass value.

  • The networking.gke.io/app-protocols annotation is not supported. Use the appProtocol field instead to achieve the same result.

  • If you use GKE Gateway with external-dns and the health state of the Gateway is unhealthy, by default, all DNS records associated with the Gateway are deleted from your DNS zones.

    Best practice:

    When running external-dns, set the policy=upsert-only flag. This configuration helps to prevent the deletion of existing DNS records.

  • If a port is removed from a Service that GKE Gateway references through a route, the Standalone NEG annotation on the Service you must also update the Standalone NEG controller on the Service to remove that port. If you don't, the NEG controller eventually stops syncing Pod endpoints for this Service. For details, see NEG Controller stops managing endpoints when port removed from Service.

Enable the Gateway API in your cluster

Before using Gateway resources in GKE, your cluster must have the Gateway API enabled.

Create a new cluster with the Gateway API enabled

GKE supports the Gateway API on Autopilot clusters starting with GKE version 1.26. If you create new Autopilot clusters on GKE 1.26 or later, the Gateway API is enabled by default. For existing clusters on GKE version 1.25 or earlier, the Gateway API is disabled by default.

If you create the cluster first without specifying a network (by using the --network flag), GKE creates the cluster in the default network. In that case, you must also create the proxy-only subnet in the default network. If you specify a network during cluster creation, ensure that you also create the proxy-only subnet in that same network.

Autopilot

Create a new GKE Autopilot cluster with the Gateway API enabled:

  gcloud container clusters create-auto CLUSTER_NAME \
      --location=CLUSTER_LOCATION \
      --release-channel=RELEASE_CHANNEL \
      --cluster-version=VERSION

Replace the following:

  • CLUSTER_NAME: the name of the cluster.
  • CLUSTER_LOCATION: the Compute Engine region or zone for the new cluster.
  • RELEASE_CHANNEL: the name of the release channel.
  • VERSION: the GKE version, which must be 1.26 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version of 1.26 or later.

When creating a cluster, you don't need to use the gcloud flag --gateway-api=standard flag.

Standard

With GKE Standard, the Gateway API is controlled by the --gateway-api flag. You can use the value standard when enabling and disabled when disabling.

Create a new VPC-native GKE cluster with the Gateway API enabled:

  gcloud container clusters create CLUSTER_NAME \
    --gateway-api=standard \
    --cluster-version=VERSION \
    --location=CLUSTER_LOCATION

Replace the following:

  • RELEASE_CHANNEL: the name of the release channel.
  • CLUSTER_NAME: the name of the cluster.
  • VERSION: the GKE version, which must be 1.24 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version of 1.24 or later.
  • CLUSTER_LOCATION: the Compute Engine region or zone for the new cluster.

The --gateway-api=standard flag instructs GKE to install the v1beta1 CRDs with the cluster.

Enable the Gateway API on an existing cluster

If you update an existing GKE Standard cluster to enable the Gateway API, make sure that the minimum requirements are met before proceeding with the update.

Ensure that your Autopilot cluster version is 1.26 or later and your Standard cluster version is 1.24 or later. To learn more about upgrading your Standard cluster version, see Standard cluster upgrades. To learn more about upgrading your Autopilot cluster version, see Autopilot cluster upgrades.

To enable the Gateway API on an existing GKE cluster (Autopilot or Standard), use the following command. When enabling Gateway on an existing cluster, it might take up to 45 minutes for the cluster to reconcile and install the CRDs.

gcloud container clusters update CLUSTER_NAME \
    --location=CLUSTER_LOCATION\
    --gateway-api=standard

Replace the following:

The --gateway-api=standard flag instructs GKE to install the v1beta1 CRDs with the cluster.

Verify your cluster

After creating or upgrading your cluster, the GKE Gateway controller automatically installs GatewayClasses. It might take a few minutes for the controller to recognize the CRDs and install the GatewayClasses.

  1. Confirm the Gateway API is enabled in the GKE control plane:

    gcloud container clusters describe CLUSTER_NAME \
  --location=CLUSTER_LOCATION \
  --format json

    The output is similar to the following. If this output is empty, re-run the cluster update command.

    "networkConfig": {
  ...
  "gatewayApiConfig": {
    "channel": "CHANNEL_STANDARD"
  },
  ...
},

  2. Confirm the GatewayClasses are installed in your cluster:

    kubectl get gatewayclass

    The output is similar to the following:

    NAME                             CONTROLLER                  ACCEPTED   AGE
gke-l7-global-external-managed   networking.gke.io/gateway   True       16h
gke-l7-regional-external-managed networking.gke.io/gateway   True       16h
gke-l7-gxlb                      networking.gke.io/gateway   True       16h
gke-l7-rilb                      networking.gke.io/gateway   True       16h

To understand the capabilities of each GatewayClass, see GatewayClass capabilities.

Only single-cluster GatewayClasses are installed automatically. To install and use the multi-cluster GatewayClasses for internal and external multi-cluster load balancing, see Enabling multi-cluster Gateways.

Deploy an internal Gateway

An internal Gateway exposes applications that are only reachable from within the VPC or networks connected to the VPC.

Deploy a regional internal Gateway

The following example shows you how to deploy a regional internal Gateway that enables efficient and secure communication between services within a specific geographic region.

Configure a proxy-only subnet

You must configure a proxy-only subnet before you create a Gateway that uses an internal Application Load Balancer. Each region of a VPC in which you use internal Application Load Balancers must have a proxy-only subnet. This subnet provides internal IP addresses to the load balancer proxies.

  1. Create a proxy-only subnet:

    gcloud compute networks subnets create SUBNET_NAME \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE \
    --region=COMPUTE_REGION \
    --network=VPC_NETWORK_NAME \
    --range=CIDR_RANGE

    Replace the following:

    • SUBNET_NAME: the name of the proxy-only subnet.
    • COMPUTE_REGION: the region of the proxy-only subnet.
    • VPC_NETWORK_NAME: the name of the VPC network in which you create this proxy-only subnet. Ensure this is the same VPC network where your GKE cluster resides and where you deploy the Gateway. This is important for seamless communication between the load balancer and your backend services.
    • CIDR_RANGE: the primary IP address range of the subnet. You must use a subnet mask no longer than /26 so that at least 64 IP addresses are available for proxies in the region. The recommended subnet mask is /23.

  2. Verify your proxy-only subnet:

    gcloud compute networks subnets describe SUBNET_NAME \
    --region=COMPUTE_REGION

    The output is similar to the following:

    ...
gatewayAddress: 10.1.1.1
ipCidrRange: 10.1.1.0/24
kind: compute#subnetwork
name: proxy-subnet
network: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/global/networks/default
privateIpGoogleAccess: false
privateIpv6GoogleAccess: DISABLE_GOOGLE_ACCESS
purpose: REGIONAL_MANAGED_PROXY
region: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION
role: ACTIVE
selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION/subnetworks/proxy-subnet
state: READY

Create a Gateway

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. To learn more about the Gateway resource, see the Gateway resource description or the API specification.

In this case, the administrator of the GKE cluster wants to create a Gateway that can be used by different teams to expose their applications internally. The administrator deploys the Gateway, and application teams deploy their Routes independently and attach them to this Gateway.

  1. Save the following Gateway manifest to a file named gateway.yaml:

    kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: internal-http
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
  - name: http
    protocol: HTTP
    port: 80

    This manifest includes the following fields:

    • gatewayClassName: gke-l7-rilb: specifies the GatewayClass that this Gateway is derived from. gke-l7-rilb corresponds to the internal Application Load Balancer.
    • port: 80: specifies that the Gateway exposes only port 80 for listening for HTTP traffic.

  2. Deploy the Gateway in your cluster:

    kubectl apply -f gateway.yaml

  3. Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.

    kubectl describe gateways.gateway.networking.k8s.io internal-http

    The output is similar to the following:

    Name:         internal-http
Namespace:    default
Spec:
  Gateway Class Name:  gke-l7-rilb
  Listeners:
    Allowed Routes:
      Kinds:
        Group:  gateway.networking.k8s.io
        Kind:   HTTPRoute
      Namespaces:
        From:  Same
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Addresses:
    Type:   IPAddress
    Value:  192.168.1.14
  Conditions:
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                NotReconciled
    Status:                False
    Type:                  Scheduled
Events:
  Type    Reason  Age                From                       Message
  ----    ------  ----               ----                       -------
  Normal  ADD     92s                networking.gke.io/gateway  test/internal-http
  Normal  UPDATE  45s (x3 over 91s)  networking.gke.io/gateway  test/internal-http
  Normal  SYNC    45s                networking.gke.io/gateway  SYNC on test/internal-http was a success

    At this point, there is a Gateway deployed in your cluster that has provisioned a load balancer and an IP address. The Gateway has no Routes, however, and so it doesn't know how it should send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404. Next, you deploy an application and Routes, which tell the Gateway how to get to application backends.

Deploy the demo applications

Application teams can deploy their applications and Routes independently from the deployment of Gateways. In some cases the application team might want to own the Gateway as well and deploy it themselves as a resource dedicated to their applications. See Route binding for different ownership models of Gateways and Routes. In this example however, the store team deploys their application and an accompanying HTTPRoute to expose their app through the internal-http Gateway created in the previous section.

The HTTPRoute resource has many configurable fields for traffic matching. For an explanation of HTTPRoute's fields, see the API specification.

  1. Deploy the store application (store-v1, store-v2, and store-german deployments) to your cluster:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml

    This creates three Deployments and three Services which are named store-v1, store-v2, and store-german.

  2. Validate that the application has deployed successfully:

    kubectl get pod

    The output is similar to the following after the application is running:

    NAME                        READY   STATUS    RESTARTS   AGE
store-german-66dcb75977-5gr2n   1/1     Running   0          38s
store-v1-65b47557df-jkjbm       1/1     Running   0          14m
store-v2-6856f59f7f-sq889       1/1     Running   0          14m

  3. Validate that the Services have been deployed:

    kubectl get service

    The output shows a Service for each store Deployment:

    NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
store-german   ClusterIP   10.48.3.183   <none>        8080/TCP   4s
store-v1       ClusterIP   10.48.2.224   <none>        8080/TCP   5s
store-v2       ClusterIP   10.48.4.48    <none>        8080/TCP   5s

Deploy the HTTPRoute

Route resources define protocol-specific rules for mapping traffic from a Gateway to Kubernetes backends. The HTTPRoute resource does HTTP and HTTPS traffic matching and filtering and is supported by all of the gke-l7 GatewayClasses.

In this section, you deploy an HTTPRoute, which programs the Gateway with the routing rules needed to reach your store application.

  1. Save the following HTTPRoute manifest to a file named store-route.yaml:

    kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  hostnames:
  - "store.example.com"
  rules:
  - backendRefs:
    - name: store-v1
      port: 8080
  - matches:
    - headers:
      - name: env
        value: canary
    backendRefs:
    - name: store-v2
      port: 8080
  - matches:
    - path:
        value: /de
    backendRefs:
    - name: store-german
      port: 8080

  2. Deploy the HTTProute in your cluster:

    kubectl apply -f store-route.yaml

    The store HTTPRoute is bound to the internal-http Gateway by using the parentRefs property. These routing rules are configured on the underlying load balancer as in this diagram:

    The routing rules configured by the store HTTPRoute

    These routing rules process HTTP traffic in the following manner:

    • Traffic to store.example.com/de goes to the Service store-german.
    • Traffic to store.example.com with the HTTP header "env: canary" goes to the Service store-v2.
    • The remaining traffic to store.example.com goes to the Service store-v1.

  3. Verify that the HTTPRoute has been deployed:

    kubectl describe httproute store

    The output is similar to the following:

    Name:         store
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1beta1
Kind:         HTTPRoute
<...>
Spec:
  Hostnames:
    store.example.com
  Parent Refs:
    Group:  gateway.networking.k8s.io
    Kind:   Gateway
    Name:   internal-http
  Rules:
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-v1
      Port:    8080
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-v2
      Port:    8080
      Weight:  1
    Matches:
      Headers:
        Name:   env
        Type:   Exact
        Value:  canary
      Path:
        Type:   PathPrefix
        Value:  /
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-german
      Port:    8080
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /de
Status:
  Parents:
    Conditions:
      Last Transition Time:  2022-11-01T04:18:52Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2022-11-01T04:18:52Z
      Message:
      Reason:                ReconciliationSucceeded
      Status:                True
      Type:                  Reconciled
    Controller Name:         networking.gke.io/gateway
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   internal-http
Events:
  Type    Reason  Age                From                   Message
  ----    ------  ----               ----                   -------
  Normal  ADD     24m                sc-gateway-controller  default/store
  Normal  SYNC    16m (x4 over 23m)  sc-gateway-controller  Bind of HTTPRoute "default/store" to ParentRef {Group:       gateway.networking.k8s.io",
  <...>

  4. Verify that the HTTPRoute is bound to the Gateway:

    kubectl describe gateway

    The output is similar to the following:

    Name:         internal-http
Namespace:    default
Labels:       <none>
<...>
Status:
  Addresses:
    Type:   IPAddress
    Value:  10.128.15.203
  Conditions:
    Last Transition Time:  2022-11-01T03:47:01Z
    Message:
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
    Last Transition Time:  2022-11-01T03:47:01Z
    Message:
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2022-11-01T03:47:01Z
      Message:
      Reason:                Ready
      Status:                True
      Type:                  Ready
    Name:                    http
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
      <...>

Send traffic to your application

Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.

  1. Retrieve the IP address from the Gateway so that you can send traffic to your application:

    kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}"

    The output is an IP address.

  2. Send traffic to this IP address from shell on a virtual machine (VM) instance with connectivity to the cluster. You can create a VM for this purpose. This is necessary because the Gateway has an internal IP address and is only accessible from within your VPC network. Because the internal-http is a regional load balancer, the client shell must be within the same region as the GKE cluster.

    Make a request to store.example.com:

    curl https://store.example.com --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    Replace GATEWAY_IP_ADDRESS with the IP address from the previous step.

    The output from the demo app shows information about the location where the app is running:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "store-v1",
  "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
  "pod_name": "store-v1-84b47c7f58-pmgmk",
  "pod_name_emoji": "💇🏼‍♀️",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-10-25T13:31:17",
  "zone": "ZONE_NAME"
}

  3. Test the path match by going to the German version of the store service at store.example.com/de:

    curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    The output confirms that the request was served by a store-german Pod:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "Gutentag!", 
  "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal",
  "pod_name": "store-german-5cb6474c55-lq5pl", 
  "pod_name_emoji": "🧞‍♀",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-10-25T13:35:37",
  "zone": "ZONE_NAME"
}

  4. Finally, use the env: canary HTTP header to send traffic to the canary version of the store Service:

    curl -H "env: canary" https://store.example.com" --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    The output confirms that the request was served by a store-v2 Pod:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "store-v2", 
  "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
  "pod_name": "store-v2-5788476cbd-s9thb", 
  "pod_name_emoji": "🦰",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-10-25T13:38:26",
  "zone": "ZONE_NAME"
}

Deploy an external Gateway

An external Gateway exposes applications that are reachable from the internet or networks outside of your VPC. The deployment is similar to an internal Gateway deployment except you must secure your applications because the Gateway is accessible to the public internet.

You have two options to create an external Gateway: a global external Gateway or a regional external Gateway.

A global external Gateway uses a Global IP address (or Anycast IP address) as the frontend of the Gateway advertised in all Google Cloud Compute regions. Clients sending traffic to this Anycast IP address are routed to the closest Google location where the IP is advertised. The global external Gateway is only available in the Premium Network Service tier.

A regional external Gateway uses a Regional IP as the frontend of the Gateway advertised only in the local Google Cloud Compute region where the regional external Gateway is deployed. Clients sending traffic to this regional IP address are routed by their local ISP and over the Internet before reaching the Google region where the IP is advertised. The regional external Gateway is only available in the Standard Network Service tier.

Deploy a global external Gateway

The following example shows you how to expose a store application with multiple certificates attached to the global external Gateway and grouped in a certificate map using Certificate Manager and an HTTPRoute.

Create a certificate map

Google recommends that you use Certificate Manager to manage certificates when you need 15 or more certificates per Gateway or you need to use wildcard certificates.

You can also secure your external Gateway using Kubernetes Secrets or Google-managed SSL certificates. For more information, see Gateway security.

In this section, you create certificates using Certificate Manager to secure the applications running on the cluster.

  1. Enable the Certificate Manager API:

    gcloud services enable certificatemanager.googleapis.com

  2. Create a certificate map:

    gcloud beta certificate-manager maps create store-example-com-map

  3. Load your Google-managed certificate and keys into a Certificate:

    gcloud beta certificate-manager certificates create store-example-com-cert \
    --certificate-file="CERTIFICATE_FILE" \
    --private-key-file="PRIVATE_KEY_FILE"

    Replace the following:

    • CERTIFICATE_FILE: the name of the new file that you choose. The file must have the extension .pem. For example, cert.pem.
    • PRIVATE_KEY_FILE: the name of your private key file.

    For more information, see Create a private key and certificate.

  4. Create a CertificateMapEntry which assigns the certificate to the certificate map:

    gcloud beta certificate-manager maps entries create store-example-com-map-entry \
    --map=store-example-com-map \
    --hostname=store.example.com \
    --certificates=store-example-com-cert

To learn how to secure a Gateway using other sources for certificates, such as Kubernetes Secrets or SSL certificates, see Secure a Gateway.

Create a Gateway

A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it uses.

To learn more about the Gateway resource, see the Gateway resource description or the API specification.

In this section, you create a Gateway. Application teams can use the Gateway to expose their applications to the internet by deploying Routes independently and attaching them securely to the Gateway.

  1. Save the following manifest to a file named gateway.yaml:

    kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: external-http
  annotations:
    networking.gke.io/certmap: store-example-com-map
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - name: https
    protocol: HTTPS
    port: 443

    This manifest describes a Gateway with the following fields:

    • gatewayClassName: gke-l7-global-external-managed: specifies the GatewayClass for this Gateway. This gateway class uses a global external Application Load Balancer.
    • protocol: HTTPS and port: 443: specify that the Gateway exposes port 443 for HTTPS traffic. These fields enables TLS.
    • networking.gke.io/certmap: store-example-com-map: specifies the name of the certificate map in Certificate Manager.

    There is no TLS section because TLS is configured with Certificate Manager using the annotation networking.gke.io/certmap.

  2. Apply the manifest to your cluster:

    kubectl apply -f gateway.yaml

    It might take a few minutes for GKE to deploy the resources.

  3. Verify that the Gateway has deployed successfully:

    kubectl describe gateway

    The output is similar to the following:

    Name:         external-http
Namespace:    default
Labels:       <none>
...
Spec:
  Gateway Class Name:  gke-l7-global-external-managed
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      https
    Port:      443
    Protocol:  HTTPS
    Tls:
      Certificate Refs:
        Group:
        Kind:   Secret
        Name:   store-example-com
      Mode:     Terminate
 ...

    This output shows that the Gateway deployed in your cluster has a load balancer and a public IP address. The Gateway has no Routes, which means it can't send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404 response. In the next section, you deploy Routes, which instruct the Gateway to send traffic to backends.

Deploy the demo applications

Application teams can deploy their applications and Routes independently from the deployment of Gateways. In some cases the application team might want to own the Gateway as well and deploy it themselves as a resource dedicated to their applications. See Route binding for different ownership models of Gateways and Routes. In this example, the store team deploys their application and an accompanying HTTPRoute to expose their app through the external-http Gateway created in the previous section.

For more information about HTTPRoute fields, see the API specification.

  1. Deploy the sample application to your cluster:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml

    This sample application creates three Deployments and three Services which are named store-v1, store-v2, and store-german.

  2. Verify that the application has deployed successfully:

    kubectl get pod

    The output is similar to the following:

    NAME                            READY   STATUS    RESTARTS   AGE
store-german-66dcb75977-5gr2n   1/1     Running   0          38s
store-v1-65b47557df-jkjbm       1/1     Running   0          14m
store-v2-6856f59f7f-sq889       1/1     Running   0          14m

  3. Verify that the Services have deployed successfully:

    kubectl get service

    The output is similar to the following:

    NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
store-german   ClusterIP   10.48.3.183   <none>        8080/TCP   4s
store-v1       ClusterIP   10.48.2.224   <none>        8080/TCP   5s
store-v2       ClusterIP   10.48.4.48    <none>        8080/TCP   5s

Create an HTTPRoute

Route resources define protocol-specific rules for mapping traffic from a Gateway to Kubernetes backends. The HTTPRoute resource does HTTP and HTTPS traffic matching and filtering and is supported by all of the gke-l7-* GatewayClasses.

In this section, you deploy an HTTPRoute, which configures the Gateway with routing rules required to reach the sample application.

  1. Save the following manifest to a file named store-route-external.yaml:

    kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store-external
spec:
  parentRefs:
  - kind: Gateway
    name: external-http
  hostnames:
  - "store.example.com"
  rules:
  - backendRefs:
    - name: store-v1
      port: 8080
  - matches:
    - headers:
      - name: env
        value: canary
    backendRefs:
    - name: store-v2
      port: 8080
  - matches:
    - path:
        value: /de
    backendRefs:
    - name: store-german
      port: 8080

    This manifest describes an HTTPRoute that references the external-http Gateway.

  2. Apply the manifest to your cluster:

    kubectl apply -f store-route-external.yaml

    The store HTTPRoute is bound to the external-http Gateway by using the parentRefs property. These following diagram shows the routing rules configured on the underlying load balancer:

    The routing rules configured by the store HTTPRoute

    The routing rules process HTTP traffic as follows:

    • Traffic to store.example.com/de routes to Service store-german.
    • Traffic to store.example.com with the HTTP header "env: canary" routes to Service store-v2.
    • The remaining traffic to store.example.com routes to Service store-v1.

  3. Verify that the HTTPRoute has been deployed:

    kubectl describe httproute store-external

    The output is similar to the following:

    Name:         store-external
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1beta1
Kind:         HTTPRoute
<...>
Spec:
  Hostnames:
    store.example.com
  Parent Refs:
    Group:  gateway.networking.k8s.io
    Kind:   Gateway
    Name:   external-http
  Rules:
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-v1
      Port:    8080
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-v2
      Port:    8080
      Weight:  1
    Matches:
      Headers:
        Name:   env
        Type:   Exact
        Value:  canary
      Path:
        Type:   PathPrefix
        Value:  /
    Backend Refs:
      Group:
      Kind:    Service
      Name:    store-german
      Port:    8080
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /de
Status:
  Parents:
    Conditions:
      Last Transition Time:  2022-11-01T05:42:31Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2022-11-01T05:43:18Z
      Message:
      Reason:                ReconciliationSucceeded
      Status:                True
      Type:                  Reconciled
    Controller Name:         networking.gke.io/gateway
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   external-http
Events:
  Type     Reason  Age    From                   Message
  ----     ------  ----   ----                   -------
  Normal   ADD     2m48s  sc-gateway-controller  default/store-external
  Normal  SYNC  61s (x3 over 2m27s)  sc-gateway-controller  Bind of HTTPRoute "default/store-external" to ParentRef Group:       "gateway.networking.k8s.io",
  ...

  4. Verify that the HTTPRoute is bound to the Gateway:

    kubectl describe gateway external-http

    The output is similar to the following:

    Name:         external-http
Namespace:    default
Labels:       <none>
<...>
Status:
  Addresses:
    Type:   IPAddress
    Value:  34.149.207.45
  Conditions:
    Last Transition Time:  2022-11-01T05:37:21Z
    Message:
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
    Last Transition Time:  2022-11-01T05:43:18Z
    Message:
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2022-11-01T05:43:18Z
      Message:
      Reason:                Ready
      Status:                True
      Type:                  Ready
    Name:                    https
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
      <...>

Send traffic to your application

Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.

  1. Get the IP address of the Gateway:

    kubectl get gateways.gateway.networking.k8s.io external-http -o=jsonpath="{.status.addresses[0].value}"

    The output is an IP address.

  2. Create a VM:

    gcloud cloud-shell ssh

  3. Send traffic to the Gateway IP address from the VM. You must set the host header manually because you do not own the example.com hostname.

    curl https://store.example.com --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    Replace GATEWAY_IP_ADDRESS with the IP address of the Gateway from the previous step.

    The output shows information from the demo app about the location where the app is running:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "store-v1",
  "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
  "pod_name": "store-v1-84b47c7f58-pmgmk",
  "pod_name_emoji": "💇🏼‍♀️",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-09-25T13:31:17",
  "zone": "us-central1-a"
}

  4. Test the path match by going to the German version of the store service at store.example.com/de:

    curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    The output confirms that the request was served by a store-german Pod:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "Gutentag!",
  "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal",
  "pod_name": "store-german-5cb6474c55-lq5pl",
  "pod_name_emoji": "🧞‍♀",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-09-25T13:35:37",
  "zone": "us-central1-a"
}

  5. Send traffic to the canary version of the store Service using the env: canary HTTP header:

    curl -H "env: canary" https://store.example.com"/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v

    The output confirms that the request was served by a store-v2 Pod:

    {
  "cluster_name": "gke1",
  "host_header": "store.example.com",
  "metadata": "store-v2",
  "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal",
  "pod_name": "store-v2-5788476cbd-s9thb",
  "pod_name_emoji": "👩🏿",
  "project_id": "gateway-demo-243723",
  "timestamp": "2022-09-25T13:38:26",
  "zone": "us-central1-a"
}

Deploy a regional external Gateway

The following example shows you how to expose a store application with multiple certificates attached to the regional external Gateway using self-managed certificates and an HTTPRoute.

Create a proxy subnet for your regional Gateway

You must configure a proxy-only subnet before you create a Gateway that uses a regional external Application Load Balancer. Each region of a VPC in which you use regional external Application Load Balancer must have an external_managed_proxy subnet. This subnet provides internal IP addresses to the load balancer proxies.

Create a certificate to secure your client traffic

You can use a certificate issued and validated by your certificate authority (CA) or create a self-signed certificate. For more information on how to create a certificate, see Store a certificate in a Kubernetes Secret.

CertificateMap or Google-managed SSL certificates are not supported with regional Gateways. Use self-managed regional SSL certificates or secrets to secure traffic between your clients and your regional Gateway. For more information about Certificate and Google Cloud load balancers, see Certificates and Google Cloud load balancers

Create a regional external HTTP(S) Gateway

  1. Create a regional static IP address for the external load balancer.

    gcloud compute addresses create IP_ADDRESS_NAME \
  --region=COMPUTE_REGION \
  --network-tier=STANDARD

    Replace the following:

    • IP_ADDRESS_NAME: the name of the new static IP address.
    • COMPUTE_REGION: The Compute Engine region where your cluster is running.

  2. Create a regional external Application Load Balancer Gateway using a self-managed certificate as follows and save the manifest as regional-gateway.yaml:

      kind: Gateway
  apiVersion: gateway.networking.k8s.io/v1beta1
  metadata:
    name: external-regional-http
  spec:
    gatewayClassName: gke-l7-regional-external-managed
    listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
        - name: store-example-com
    addresses:
    - type: NamedAddress
      value: IP_ADDRESS_NAME

  3. Apply the regional-gateway manifest:

      kubectl apply -f regional-gateway.yaml

  4. Verify your configuration.

      kubectl get gateway

    The output is similar to the following:

    NAME            CLASS                              ADDRESS         READY   AGE
external-http   gke-l7-regional-external-managed   35.118.32.224   True    49s

    To get more details, use a describe command:

    kubectl describe gateway

    The output is similar to the following:

    Name:         external-regional-http
Namespace:    default
Labels:       <none>
...
Spec:
  Gateway Class Name:  gke-l7-regional-external-managed
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      https
    Port:      443
    Protocol:  HTTPS
    Tls:
      Certificate Refs:
        Group:
        Kind:   Secret
        Name:   store-example-com
      Mode:     Terminate
  ...

Deploy the demo application

You can deploy your applications and routes independently from the deployment of Gateways.

For more information on how to deploy the demo applications, see Deploy the demo applications.

Create an HTTPRoute

You must create an HTTPRoute to do HTTP and HTTPS traffic matching and filtering.

Send Traffic to your application

After you have deployed your application and created HTTPRoutes, you can pass traffic to your application.

For more information on how to send traffic to your application, see Send traffic to your application.

Use shared Gateways

The Gateway APIs use separate resources, Gateways and Route resources, to deploy load balancers and routing rules. This differs from Ingress, which combines everything in one resource. By splitting responsibility among resources, Gateway enables the load balancer and its routing rules to be deployed separately and to be deployed by different users or teams. This enables Gateways to become shared Gateways that attach with many different Routes that can be fully owned and managed by independent teams, even across different namespaces.

Deploy routes against a shared Gateway

This example builds on the internal-http Gateway deployed in Deploy an internal Gateway.

In this example, the site team deploys their application, Services, and an HTTPRoute to match traffic from the Gateway to those Services.

  1. Deploy the example application:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/site.yaml

  2. Save the following manifest to a file named site-route-internal.yaml:

    kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: site-internal
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  hostnames:
  - "site.example.com"
  rules:
  - backendRefs:
    - name: site-v1
      port: 8080

    This manifest describes an HTTPRoute that matches all traffic for site.example.com and routes it to the site-v1 Service.

  3. Apply the manifest to your cluster:

    kubectl apply -f site-route-internal.yaml

  4. Verify that the HTTPRoute is attached to the Gateway:

    kubectl describe httproute.gateway.networking.k8s.io site-internal

    The output is similar to the following:

    Status:
  Parents:
    Conditions:
      Last Transition Time:  2023-01-09T15:05:43Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2023-01-09T15:05:43Z
      Message:
      Reason:                ReconciliationSucceeded
      Status:                True
      Type:                  Reconciled
    Controller Name:         networking.gke.io/gateway
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   internal-http
      ...

    If the Accepted condition for the Gateway is True, the HTTPRoute has successfully bound to the Gateway. To learn more about the Status field, see route status.

  5. Verify that traffic to the Gateway is routed correctly:

    curl -H "host: site.example.com" GATEWAY_IP_ADDRESS
curl -H "host: store.example.com" GATEWAY_IP_ADDRESS

    Replace GATEWAY_IP_ADDRESS with the IP address of the Gateway.

    You must use a virtual machine (VM) in the same VPC as the Gateway.

    The output is similar to the following:

    {
  "cluster_name": "CLUSTER_NAME",
  "host_header": "site.example.com",
  "metadata": "site-v1",
  "pod_name": "site-v1-5d64fc4d7d-fz6f6",
  "pod_name_emoji": "👩🏼‍🍳",
  "project_id": "PROJECT_ID",
  "timestamp": "2022-11-02T19:07:01",
  "zone": "ZONE_NAME"
}
...
{
  "cluster_name": "CLUSTER_NAME",
  "host_header": "store.example.com",
  "metadata": "store-v1",
  "pod_name": "store-v1-6d8d58d78-vz8pn",
  "pod_name_emoji": "🧝🏻‍♂️",
  "project_id": "PROJECT_ID",
  "timestamp": "2022-11-02T19:07:01",
  "zone": "ZONE_NAME"
}

Configure the Gateway default backend

All of the gke-l7-* GatewayClasses return HTTP 404 to unmatched traffic. You can configure the default backend using an explicit default Route that sends unmatched traffic to a user-provided Service.

Gateways are configured to handle error codes like 404 (Not Found) and 500 (Server Error), even without explicit backend definitions. The default behavior may vary between Gateway implementations. For greater control over error handling, consider configuring custom backends.

The following HTTPRoute is an example of how to customize the default backend. If you apply an HTTPRoute similar to the following, it takes precedence over the implicit default backend:

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: custom-default-backend
spec:
  parentRefs:
  - kind: Gateway
    name: my-internal-gateway
  rules:
  - backendRefs:
    - name: my-custom-default-backend-service
      port: 8080

This HTTPRoute matches all traffic from a particular Gateway. You can only have one such rule for each Gateway or else the rules conflict and precedence ordering applies.

You can use a default backend to prevent someone from creating a default route Backend that routes all Gateway traffic. An explicit HTTPRoute always takes precedence over new HTTPRoutes with conflicting routing rules.

Configure a static IP address for a Gateway

Every Gateway has an IP address it uses to listen for traffic. If you do not specify an IP address on the Gateway, then the Gateway controller automatically provides an IP address. You can also create a static IP address so that the IP address exists independent of the Gateway lifecycle.

After a Gateway is deployed, its IP address shows in the status field:

kind: Gateway
...
status:
  addresses:
    - value: 10.15.32.3

Depending on the GatewayClass, the IP address is allocated from the following subnets:

GatewayClasses Default IP Address Pool
  • gke-l7-rilb
  • gke-l7-rilb-mc
    		•  Regional private IP addresses from the primary node IPv4/IPv6 address range
  • gke-l7-regional-external-managed
  • gke-l7-regional-external-managed-mc
    		•  Regional public IP addresses from Google's regional external IPv4/IPv6 ranges
  • gke-l7-global-external-managed
  • gke-l7-global-external-managed-mc
  • gke-l7-gxlb
  • gke-l7-gxlb-mc
    		•  Global public IP addresses from Google's global external IPv4/IPv6 ranges

    The field addresses.NamedAddress lets you specify an IP address independently of the Gateway. You can create a static IP address resource prior to Gateway deployment and the resource is referenced by the NamedAddress. You can reuse the static IP address even if the Gateway is deleted.

    Use a named IP address

    You can configure an IPv4 or IPv6 address by specifying a NamedAddress. You must provision a static IP address before you create a Gateway.

    1. Create a static IP address resource:

      gcloud compute addresses create IP_ADDRESS_NAME \
    --purpose=SHARED_LOADBALANCER_VIP \
    --region=COMPUTE_REGION \
    --subnet=SUBNET \
    --project=PROJECT_ID

      Replace the following:

      • IP_ADDRESS_NAME: the name of the new static IP address
      • COMPUTE_REGION: for regional Gateways, the Compute Engine region where your cluster is running. This flag is not needed for global, external Gateways.
      • SUBNET: the subnet for the IP address. This flag is not needed for global, external Gateways.
      • PROJECT_ID: the project where your GKE cluster is running.

    2. Save the following manifest to a file named named-ip-gateway.yaml:

      kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: internal-http
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
  - name: http
    protocol: HTTP
    port: 80
  addresses:
  - type: NamedAddress
    value: IP_ADDRESS_NAME

      This manifest describes a Gateway that references the named IP address.

    3. Apply the manifest to your cluster:

      kubectl apply -f named-ip-gateway.yaml

    4. Verify your Gateway IP address:

      kubectl describe gateway internal-http

      The output is similar to the following:

      Name:         internal-http
Namespace:    default
Labels:       <none>
...
Spec:
  Addresses:
    Type:              NamedAddress
    Value:             IP_ADDRESS_NAME
  Gateway Class Name:  gke-l7-rilb
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Addresses:
    Type:   IPAddress
    Value:  10.15.32.103

    Configure HTTP-to-HTTPS redirects

    Cloud Load Balancing offers HTTP to HTTPS redirect functionality. An external Application Load Balancer redirects unencrypted HTTP requests to an HTTPS load balancer that uses the same IP address. When you create a Gateway with HTTP-to-HTTPS redirects enabled, both of these load balancers are created automatically. Requests to the external IP address of the Gateway on port 80 are automatically redirected to the same external IP address on port 443.

    By default, HTTP to HTTPS redirects are not defined on the Gateway.

    To redirect HTTP traffic to HTTPS, configure a Gateway to handle both HTTP and HTTPS traffic. If you disable either HTTP or HTTPS, the Gateway does not redirect traffic.

    The following example shows you how you can use HTTP-to-HTTPS redirect as a means to ensure that traffic from your clients going to your web applications is always being redirected to a secure page.

    HTTP-to-HTTPS redirects are not supported with the gke-l7-gxlb and gke-l7-gxlb-mc GatewayClasses. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.

    Redirect HTTP traffic from an infrastructure namespace

    In some cases, there isn't a clear distinction between the infrastructure or platform admin team and the application teams and preventing misuse of the Gateway can become a challenge.

    The following example further restricts the use of the HTTP listener to prevent unintentional use of non-secure protocol from the application teams. This example configures the Gateway to allow an HTTPRoute to use the HTTP listener only if the route is in a given namespace (http-redirect) while it opens the HTTPS listener to all namespaces. You can restrict the http-redirect namespace using Kubernetes RBAC so that application teams cannot create an HTTPRoute in this namespace by mistake.

    1. Create the namespace of a Gateway. Save the manifest as gateway-namespace.yaml:

      apiVersion: v1
kind: Namespace
metadata:
  name: gateway-infra

    2. Apply the manifest:

      kubectl apply -f gateway-namespace.yaml

    3. Create the namespace of a Gateway and save the manifest as redirect-namespace.yaml:

      apiVersion: v1
kind: Namespace
metadata:
  name: http-redirect
  labels:
    otherInfra: httpToHttps
      • Specific labels are set for this namespace.

    4. Apply the manifest:

      kubectl apply -f redirect-namespace.yaml

    5. To restrict the http listener usage, create a Gateway using the following manifest. Save the manifest as external-gateway.yaml:

      kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: external-http
  namespace: gateway-infra
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      kinds:
      - kind: HTTPRoute
      namespaces:
        from: selector
        selector:
          matchLabels:
            otherInfra: httpToHttps
  - name: https
    protocol: HTTPS
    port: 443
    allowedRoutes:
      kinds:
      - kind: HTTPRoute
      namespaces:
        from: All
    tls:
      mode: Terminate
      options:
        networking.gke.io/pre-shared-certs: store-example-com
  ```

      • The namespace field specifies that the Gateway is created in the gateway-infra namespace.

      • The namespaces field in the allowedRoutes section restricts the http listener to the namespace matching the label otherInfra: httpToHttps.

    6. Apply the manifest:

      kubectl apply -f external-gateway.yaml

    7. To force the HTTPS redirect, create a default HTTPRoute using the following manifest. Save the manifest as http-redirect.yaml:

      kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: redirect
  namespace: http-redirect
spec:
  parentRefs:
  - namespace: gateway-infra
    name: external-http
    sectionName: http
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
      • The sectionName field instructs the Gateway to match only on the http listener. The RequestRedirect filter forces the redirection to the https listener.

    8. Apply the manifest:

      kubectl apply -f http-redirect.yaml

    9. Create a Service for an application using the following manifest. Save the manifest as service-deployment.yaml:

      apiVersion: v1
kind: Service
metadata:
  name: store-v1
spec:
  selector:
    app: store
    version: v1
  ports:
  - port: 8080
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: store-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: store
      version: v1
  template:
    metadata:
      labels:
        app: store
        version: v1
    spec:
      containers:
      - name: whereami
        image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.20
        ports:
        - containerPort: 8080
        env:
        - name: METADATA
          value: "store-v1"

    10. Apply the manifest:

      kubectl apply -f service-deployment.yaml

    11. Create an HTTPRoute for an application that only allows HTTPS using the following manifest. Save the manifest as http-route.yaml:

      kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store-external
  labels:
    gateway: external-http
spec:
  parentRefs:
  - name: external-http
    namespace: gateway-infra
    sectionName: https
  hostnames:
  - "store.example.com"
  rules:
  - backendRefs:
    - name: store-v1
      port: 8080

    12. Apply the manifest:

      kubectl apply -f http-route.yaml

    Configure path redirects and URL rewrites

    Path redirects involve redirecting an incoming request from one URL path to another. Path redirects let you change the structure of the URL when you need to handle outdated or deprecated URLs.

    URL rewrites help modify the incoming URL before processing it on the server. It allows you to change the structure or format of the URL without actually changing the underlying content or file structure. URL rewriting is beneficial for creating user-friendly and SEO-friendly URLs that are easy to remember and understand. By default, path redirects and URL rewrites are not configured, you need to explicitly configure those redirects or rewrites using a filter in your HTTPRoute.

    GKE Gateway supports path redirects and URL rewrites. For more information, see HTTP path redirects and rewrites.

    Configure path redirects

    You can configure path redirects to either replace the entire path or only a prefix in the URL.

    Replace entire path

    1. To replace an entire path, configure a filter in an HTTPRoute that replaces any URL that contains the prefix /any-path in the URL path by the strict value /new-path.

    2. Create an HTTPRoute manifest as follows and name it as store.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: store
  spec:
    parentRefs:
      - kind: Gateway
        name: external-http
    hostnames:
    - store.example.com
    rules:
    - matches:
      - path:
          type: PathPrefix
          value: /any-path
      filters:
      - type: RequestRedirect
        requestRedirect:
          path:
            type: ReplaceFullPath
            replaceFullPath: /new-path
          statusCode: 302

      For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/... should be redirected to a new location, https://store.example.com/new-path/ (strict).

    3. Apply the manifest:

      kubectl apply -f store.yaml

    This routing rule follows a strict redirection rule, which means that the browser does not attempt to cache the redirect, instead, redirects to the latest version.

    Replace a prefix only

    1. To replace a prefix only, configure a filter in an HTTPRoute that replaces any URL that contains the prefix /any-prefix in the URL path by the strict value /new-prefix.

    2. Create an HTTPRoute manifest as follows and name it as store.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: store
spec:
  parentRefs:
    - kind: Gateway
      name: external-http
  hostnames:
  - store.example.com
  rules:
  - matches:
      - path:
          type: PathPrefix
          value: /any-prefix
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /new-prefix
        statusCode: 302

      For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/v1/... should be redirected to a new location, https://store.example.com/new-path/v1/... (only).

    3. Apply the manifest:

        kubectl apply -f store.yaml

    This routing rule follows the only redirection rule, which ensures that the browser always redirects you to the same intended page.

    Configure URL rewrites

    Set URL rewrites to change the way a URL appears to users. You can use URL rewrites to make URLs more user-friendly, to improve SEO, or to redirect users to a new page.

    Rewrite the entire hostname

    To rewrite the entire hostname:

    1. Configure a filter in an HTTPRoute that instructs the Gateway to replace the Host information in the request header from www.example.com to store.example.com before forwarding the request to the backend service.

    2. Create an HTTPRoute manifest as follows and name it as www.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: www
  spec:
    parentRefs:
      - kind: Gateway
        name: external-http
    hostnames:
    - www.example.com
    rules:
    - filters:
      - type: URLRewrite
        urlRewrite:
          hostname: store.example.com
      backendRefs:
      - name: store-v1
        port: 8080

      For example, with the above configuration, any request to https://www.example.com is forwarded to the backend service with the Host: store.example.com header, instead of Host: www.example.com.

    3. Apply the manifest:

        kubectl apply -f www.yaml

    Rewrite using path modifiers

    You can combine rewrites with path modifiers to provide advanced URL and path modifications before relaying the request to the backend service.

    To rewrite using path modifiers:

    1. Configure a filter in an HTTPRoute that instructs the Gateway to replace the 'Host' information in the request header from www.example.comto store.example.com and replace the value /store by / before forwarding the request to the backend service.

    2. Create an HTTPRoute manifest as follows and name it as www.yaml:

        apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: www
  spec:
    parentRefs:
      - kind: Gateway
        name: external-http
    hostnames:
    - www.example.com
    rules:
    - matches:
      - path:
          type: PathPrefix
          value: /store
      filters:
      - type: URLRewrite
        urlRewrite:
          hostname: store.example.com
          path:
            type: ReplacePrefixMatch
            replacePrefixMatch: /de
      backendRefs:
      - name: store-german
        port: 8080

      For example, with the above configuration, any request to https://www.example.com/store/... is forwarded to the backend service with Host: store.example.com in the request header (instead of Host: www.example.com) and the /store is rewritten to /de.

    3. Apply the manifest:

      kubectl apply -f www.yaml

    Verify your configuration

    To verify that the filter was applied after creating your HTTPRoute with URL rewrite or path redirects filters, do the following:

    kubectl get httproute www -o yaml

    The output is similar to the following:

      apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"gateway.networking.k8s.io/v1beta1","kind":"HTTPRoute","metadata":{"annotations":{},"name":"www","namespace":"default"},"spec":{"hostnames":["www.example.com"],"parentRefs":[{"kind":"Gateway","name":"external-http"}],"rules":[{"backendRefs":[{"name":"store-german","port":8080}],"filters":[{"type":"URLRewrite","urlRewrite":{"hostname":"store.example.com","path":{"replacePrefixMatch":"/de","type":"ReplacePrefixMatch"}}}],"matches":[{"path":{"type":"PathPrefix","value":"/store"}}]}]}}
    creationTimestamp: "2023-06-22T01:00:42Z"
    generation: 3
    name: www
    namespace: default
    resourceVersion: "51268631"
    uid: e516493e-806d-44d6-ae0d-1c9ff25682cf
  spec:
    hostnames:
    - www.example.com
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: external-http
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: store-german
        port: 8080
        weight: 1
      filters:
      - type: URLRewrite
        urlRewrite:
          hostname: store.example.com
          path:
            replacePrefixMatch: /de
            type: ReplacePrefixMatch
      matches:
      - path:
          type: PathPrefix
          value: /store
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2023-06-22T01:11:26Z"
        message: ""
        observedGeneration: 2
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2023-06-22T01:11:26Z"
        message: ""
        observedGeneration: 2
        reason: ReconciliationSucceeded
        status: "True"
        type: Reconciled
      controllerName: networking.gke.io/gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: external-http

    To get more details, use the describe command:

    kubectl describe httproute

    Configure custom request and response headers

    Custom request and response headers let you specify additional headers to HTTP(S) requests and responses. Depending on the information detected by the load balancer, these headers can include the following information:

    • Latency to the client
    • Geographic location of the client's IP address
    • Parameters of the TLS connection

    By default, there are no custom headers added to the request sent/received to/from your backend services, you need to explicitly configure custom headers using a filter in your HTTPRoute.

    You can configure custom headers by adding a filter section in your HTTPRoute's rules as follows:

    Configure custom request headers

    Create a HTTPRoute manifest with a RequestHeaderModifier filter and save it as http-route-request.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: store
  spec:
    <...>
    rules:
        filters:
          - type: RequestHeaderModifier
            requestHeaderModifier:
              <...>

    Apply the manifest:

      kubectl apply -f http-route-request.yaml

    Configure custom response headers

    Create a HTTPRoute manifest with a ResponseHeaderModifier filter and save it as http-route-response.yaml:

    apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: store
spec:
  <...>
  rules:
      filters:
        - type: ResponseHeaderModifier
          responseHeaderModifier:
            <...>

    Apply the manifest:

      kubectl apply -f http-route-response.yaml

    You can add, set, and remove headers as described in the Gateway API implementation. You can configure your HTTPRoute with a custom header using Google Cloud supported variables.

    Example 1:

    To configure an HTTPRoute that adds client location information to the HTTP request before sending it to the backend service, create a HTTPRoute manifest and name it as external-http-request.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: store
  spec:
    parentRefs:
      - kind: Gateway
        name: external-http
    hostnames:
    - store.example.com
    rules:
      - matches:
        - path:
            type: PathPrefix
            value: /fr
        filters:
          - type: RequestHeaderModifier
            requestHeaderModifier:
              add:
                - name: X-Client-Geo-Location
                  value: "{client_region},{client_city}"
        backendRefs:
          - name: store-french
            port: 8080

    For example, for clients located in Strasbourg, France, the Gateway adds a header as X-Client-Geo-Location:FR,Strasbourg.

    Example 2:

    To configure an HTTPRoute that adds a custom response header to support HTTP Strict Transport Security, create a HTTPRoute manifest and name it as external-http-response.yaml:

      apiVersion: gateway.networking.k8s.io/v1beta1
  kind: HTTPRoute
  metadata:
    name: store
  spec:
    parentRefs:
      - kind: Gateway
        name: external-http
    hostnames:
    - store.example.com
    rules:
      - matches:
        - path:
            type: PathPrefix
            value: /de
        filters:
          - type: ResponseHeaderModifier
            responseHeaderModifier:
              add:
                - name: Strict-Transport-Security
                  value: max-age=63072000
        backendRefs:
          - name: store-german
            port: 8080

    Verify your configuration

    1. To verify your configuration after configuring custom request and response headers, do the following:

        kubectl get httproute

      The output is similar to the following:

        NAME    HOSTNAMES               AGE
  store   ["store.example.com"]   4d23h

    2. To get more details, use the describe command:

        kubectl describe httproute

      The output is similar to the following:

        Name:         store
  Namespace:    default
  Labels:       <none>
  Annotations:  <none>
  API Version:  gateway.networking.k8s.io/v1beta1
  Kind:         HTTPRoute
  Metadata:
    Creation Timestamp:  2023-05-27T00:51:01Z
    Generation:          5
    Resource Version:    25418887
    UID:                 2e07a1b8-420b-41b4-acd1-cecbfcd39f42
  Spec:
    Hostnames:
      store.example.com
    Parent Refs:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   external-http
    Rules:
      Backend Refs:
        Group:
        Kind:    Service
        Name:    store-v1
        Port:    8080
        Weight:  1
      Matches:
        Path:
          Type:   PathPrefix
          Value:  /
      Backend Refs:
        Group:
        Kind:    Service
        Name:    store-v2
        Port:    8080
        Weight:  1
      Matches:
        Headers:
          Name:   env
          Type:   Exact
          Value:  canary
        Path:
          Type:   PathPrefix
          Value:  /
      Backend Refs:
        Group:
        Kind:    Service
        Name:    store-german
        Port:    8080
        Weight:  1
      Filters:
        Request Header Modifier:
          Add:
            Name:   X-Client-Geo-Location
            Value:  {client_region},{client_city}
        Type:       RequestHeaderModifier
      Matches:
        Path:
          Type:   PathPrefix
          Value:  /de
  Status:
    <...>

    Route status

    HTTPRoute resources emit conditions and events to help users understand if a HTTPRoute has successfully bound with one or more Gateways or if it was rejected.

    HTTPRoute conditions

    HTTPRoute conditions indicate the status of the Route and the Gateways it is bound to. Because a Route can be bound to multiple Gateways, this is a list of Gateways and the individual conditions between the Route and each Gateway.

    • Accepted=True indicates that the HTTPRoute is successfully bound to a Gateway.
    • Accepted=False indicates that the HTTPRoute has been rejected from binding with this Gateway.

    If there are no Gateways listed under the Gateway bindings heading, then your HTTPRoute labels and Gateway label selectors might not match. This can occur if your Route is not being selected by any Gateways.

    HTTPRoute events

    HTTPRoute events provide details about the status of the HTTPRoute. Events are grouped by the following reasons:

    • ADD events are triggered by a resource being added.
    • UPDATE events are triggered by a resource being updated.
    • SYNC events are triggered by periodic reconciliation.

    Route merging, precedence, and validation

    Route precedence

    The Gateway API defines strict precedence rules for how traffic is matched by Routes that have overlapping routing rules. The precedence between two overlapping HTTPRoutes is as follows:

    1. Hostname merge: The longest/most specific hostname match.
    2. Path merge: The longest/most specific path match.
    3. Header merge: The largest number of HTTP headers that match.
    4. Conflict: If the previous three rules don't establish precedence, then precedence goes to the HTTPRoute resource with the oldest timestamp.

    Route merging

    For gke-l7 GatewayClasses, all HTTPRoutes for a given Gateway are merged into the same URL map resource. How the HTTPRoutes are merged together depends on the type of overlap between HTTPRoutes. The HTTPRoute from the earlier example can be split into three separate HTTPRoutes to illustrate route merging and precedence:

    1. Route merge: All three HTTPRoutes attach with the same internal-http Gateway, so they are merged together.
    2. Hostname merge: All three Routes match for store.example.com, so their hostname rules are merged.
    3. Path merge: store-german-route has a more specific path /de, so this is not merged further. store-v1-route and store-v2-route both match on the same /* path as well, so they are merged on the path.
    4. Header merge: store-v2-route has a more specific set of HTTP header matches than store-v1-route, so they are not merged further.
    5. Conflict: Because the Routes are able to be merged on hostname, path, and headers, there are no conflicts, and all of the routing rules are applied to traffic.

    The single HTTPRoute used in the earlier example are equivalent to these three separate routes:

    kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store-v1-route
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  hostnames:
  - "store.example.com"
  rules:
  - backendRefs:
    - kind: Service
      name: store-v1
      port: 8080
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store-v2-route
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  hostnames:
  - "store.example.com"
  rules:
  - matches:
    - headers:
      - type: Exact
        name: env
        value: canary
    backendRefs:
    - kind: Service
      name: store-v2
      port: 8080
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store-german-route
spec:
  parentRefs:
  - kind: Gateway
    name: internal-http
  hostnames:
  - "store.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /de
    backendRefs:
    - kind: Service
      name: store-german
      port: 8080

    Kubernetes Gateways and Istio Gateways

    Note that the Kubernetes Gateway API and the Istio API both have a resource named Gateway. While they perform similar functions, they are not the same resource. If you are using Istio and the Gateway API in the same Kubernetes cluster, these names overlap when using kubectl on the command line. kubectl get gateway might return the Kubernetes Gateway resources and not the Istio Gateway resources or vice versa.

    $ kubectl api-resources
NAME       SHORTNAMES   APIGROUP                       NAMESPACED   KIND
gateways   gw           networking.istio.io/v1beta1    true         Gateway
gateways   gtw          networking.k8s.io/v1beta1      true         Gateway

    If you are using Istio and upgrade to GKE 1.20 and later it is recommended to start using the Gateway resource shortname or specify the API group. The shortname for a Kubernetes Gateway is gtw and the shortname for an Istio Gateway is gw. The following commands return the Kubernetes Gateway and Istio Gateway resources respectively.

    # Kubernetes Gateway
$ kubectl get gtw
NAME                        CLASS
multi-cluster-gateway       gke-l7-global-external-managed-mc

$ kubectl get gateway.networking.x-k8s.io
NAME                        CLASS
multi-cluster-gateway       gke-l7-global-external-managed-mc

# Istio Gateway
$ kubectl get gw
NAME               AGE
bookinfo-gateway   64m

$ kubectl get gateway.networking.istio.io
NAME               AGE
bookinfo-gateway   64m

    Troubleshooting

    Proxy-only subnet missing in the region

    Symptom:

    The following issue might occur when you create a regional Gateway (internal or external):

    generic::invalid_argument: error ensuring load balancer: Insert: Invalid value for field 'resource.target': 'regions/[REGION_NAME]/targetHttpProxies/gkegw-x5vt-default-internal-http-[ID]'. A reserved managed proxy subnetwork with purpose REGIONAL_MANAGED_PROXY is required.

    Reason:

    This error message indicates that no proxy-only subnet exists in the region for your Gateway.

    Workaround:

    To resolve this issue, configure a proxy-only subnet.

    Proxy-only subnet already exists in the region with the wrong purpose

    Symptom:

    The following issue might occur when you create a proxy-only subnet for your regional Gateway (internal or external):

    ERROR: (gcloud.compute.networks.subnets.create) Could not fetch resource:
 - The resource 'projects/[PROJECT_NAME]/regions/[REGION_NAME]/subnetworks/[PROXY_ONLY_SUBNET_NAME]' already exists

    Reason:

    This error message indicates that you attempted to create a regional proxy-only subnet in a region that already has a proxy-only subnet.

    Workaround:

    To resolve this issue, use the following steps:

    1. Check that a proxy-only subnet already exists in the region and verify that it has the correct purpose:

      1. List your subnets to find which one is the proxy-only subnet in the region:

        gcloud compute networks subnets list --regions=COMPUTE_REGION

        Replace COMPUTE_REGION with the Compute Engine region where you want to create your regional Gateway.

      2. Describe your proxy-only subnet in the region to find its purpose:

        gcloud compute networks subnets describe PROXY_ONLY_SUBNET \
    --region COMPUTE_REGION | grep -E 'name|purpose'

        Replace PROXY_ONLY_SUBNET with the proxy-only subnet.

      GKE Gateway only supports REGIONAL_MANAGED_PROXY proxy-only subnets for regional Gateways (internal or regional).

    2. If the existing proxy-only subnet in the region was created with an INTERNAL_HTTPS_LOAD_BALANCER purpose, migrate its purpose to REGIONAL_MANAGED_PROXY.

    No healthy upstream

    Symptom:

    The following issue might occur when you create a Gateway but cannot access the backend services (503 response code):

    no healthy upstream

    Reason:

    This error message indicates that the health check prober cannot find healthy backend services. It is possible that your backend services are healthy but you might need to customize the health checks.

    Workaround:

    To resolve this issue, customize your health check based on your application's requirements (for example, /health) using a HealthCheckPolicy.

    What's next