This page describes how to deploy Kubernetes Gateway resources for load balancing ingress traffic to a single Google Kubernetes Engine (GKE) cluster.
For deploying Gateways to load balance ingress traffic across multiple clusters (or fleet), see Deploying Multi-Cluster Gateways.Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
GKE Gateway controller requirements
- For Standard, GKE version 1.24 or later.
- For Autopilot, GKE version 1.26 or later.
- Google Cloud CLI version 407.0.0 or later.
- The Gateway API is supported on VPC-native clusters only.
- If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
- Your cluster must have the
HttpLoadBalancing
add-on enabled. - If you are using Istio, you must upgrade Istio to one of the following
versions:
- 1.15.2 or later
- 1.14.5 or later
- 1.13.9 or later.
- If you are using Shared VPC, then in the host project, you need to assign the
Compute Network User
role to the GKE Service account for the service project.
Restrictions and limitations
When using GKE Gateway, understand the following limitations and restrictions:
GKE GatewayClasses support different capabilities depending on the load balancer they use. To learn more about the different features supported with each GatewayClass, see GatewayClass capabilities.
Best practice: For optimal performance, limit the number of Gateways to a maximum of 100. Exceeding this limit can affect performance or result in increased latency.
You cannot use a FrontendConfig or a BackendConfig to configure a Gateway. You must use a Policy.
GKE Gateway behaves differently than Ingress, in that Gateway does not infer health check parameters. If your Service does not return 200 for requests to
GET /
, or you have other tuned pod readiness checks, you need to configure a HealthCheckPolicy for your service.You cannot specify a port number directly in the hostname (for example, web.example.com:80) for traffic routing.
You can view the load balancer resources that GKE creates for Gateways in the Google Cloud console, but these resources don't reference the Gateway or GKE cluster they are attached to.
- You can't automatically generate a Google-managed SSL certificate with Gateways but you can manually create and reference a Google-managed SSL certificate. For more information, see Secure a Gateway.
HTTPRoute is the only Route type supported. TCPRoutes, UDPRoutes, and TLSRoutes are not supported. To see a list of fields the GKE Gateway controller supports, see GatewayClass capabilities.
Custom request and response headers with Gateway or path redirects and URL rewrites with Gateway is only available on GKE version 1.27 or later.
- For custom request and response headers with Gateway and path redirects and
URL rewrites with Gateway, the GatewayClass
gke-l7-gxlb
is not supported.
When configuring HTTPRoute custom request and response headers, the following Google Cloud variables are not supported:
cdn_cache_id
(Cloud CDN is not supported with GKE Gateway)cdn_cache_status
(Cloud CDN is not supported with GKE Gateway)origin_request_header
(CORS policies are not supported with GKE Gateway)
GKE Gateway does not support the Cloud CDN load balancing feature.
Mutual TLS custom headers are not supported (mTLS with GKE Gateway is not supported)
Google Cloud classic Application Load Balancer limitations apply to the GKE Gateway. In addition, you can't configure a custom Host response header in the backend service.
Path redirects and URL rewrites are mutually exclusive, you can't use both filters at the same time in the same rules.
Redirecting traffic to a different port is not supported with Cloud Load Balancing. To see the list of fields the GKE Gateway controller supports, see GatewayClass capabilities.
GKE Gateway does not support Wildcards, regular expressions, and dynamic URLs.
If you specify a Gateway with a regional external gateway class, the controller provisions an internal IP address instead of the external address. To learn how to use a named address with the regional external Application Load Balancer, see deploy a regional external Gateway.
Gateway utilizes Standalone NEGs for provisioning Network Endpoint Groups. To ensure that the Gateway controller properly reconciles the load balancer configuration, you cannot modify the
cloud.google.com/neg
annotation for a Service that is part of the Gateway.GKE Gateway does not support referencing a Service that is also referenced by a GKE Ingress.
When a
Gateway
is configured to provision an IP address, changing theGateway.spec.gatewayClass
is not supported. To ensure that the Gateway controller properly reconciles the load balancer, delete the existing Gateway and re-deploy the manifest with the updatedgatewayClass
value.The
networking.gke.io/app-protocols
annotation is not supported. Use theappProtocol
field instead to achieve the same result.If you use GKE Gateway with
external-dns
and the health state of the Gateway is unhealthy, by default, all DNS records associated with the Gateway are deleted from your DNS zones.Best practice: When running
external-dns
, set thepolicy=upsert-only
flag. This configuration helps to prevent the deletion of existing DNS records.If a port is removed from a
Service
that GKE Gateway references through a route, the Standalone NEG annotation on the Service you must also update the Standalone NEG controller on the Service to remove that port. If you don't, the NEG controller eventually stops syncing Pod endpoints for this Service. For details, see NEG Controller stops managing endpoints when port removed from Service.
Enable the Gateway API in your cluster
Before using Gateway resources in GKE, your cluster must have the Gateway API enabled.
Create a new cluster with the Gateway API enabled
GKE supports the Gateway API on Autopilot clusters starting with GKE version 1.26. If you create new Autopilot clusters on GKE 1.26 or later, the Gateway API is enabled by default. For existing clusters on GKE version 1.25 or earlier, the Gateway API is disabled by default.
If you create the cluster first without specifying a network (by using the
--network
flag), GKE creates the cluster in the default
network. In that case, you must also create the proxy-only
subnet
in the default network. If you specify a network during cluster creation, ensure
that you also create the proxy-only subnet in that same network.
Autopilot
Create a new GKE Autopilot cluster with the Gateway API enabled:
gcloud container clusters create-auto CLUSTER_NAME \
--location=CLUSTER_LOCATION \
--release-channel=RELEASE_CHANNEL \
--cluster-version=VERSION
Replace the following:
CLUSTER_NAME
: the name of the cluster.CLUSTER_LOCATION
: the Compute Engine region or zone for the new cluster.RELEASE_CHANNEL
: the name of the release channel.VERSION
: the GKE version, which must be 1.26 or later. You can also use the--release-channel
flag to select a release channel. The release channel must have a default version of 1.26 or later.
When creating a cluster, you don't need to use the gcloud flag --gateway-api=standard
flag.
Standard
With GKE Standard, the Gateway API is controlled by the
--gateway-api
flag. You can use the value standard when enabling and disabled when disabling.
Create a new VPC-native GKE cluster with the Gateway API enabled:
gcloud container clusters create CLUSTER_NAME \
--gateway-api=standard \
--cluster-version=VERSION \
--location=CLUSTER_LOCATION
Replace the following:
RELEASE_CHANNEL
: the name of the release channel.CLUSTER_NAME
: the name of the cluster.VERSION
: the GKE version, which must be 1.24 or later. You can also use the--release-channel
flag to select a release channel. The release channel must have a default version of 1.24 or later.CLUSTER_LOCATION
: the Compute Engine region or zone for the new cluster.
The --gateway-api=standard
flag instructs GKE to install the
v1beta1
CRDs with the cluster.
Enable the Gateway API on an existing cluster
If you update an existing GKE Standard cluster to enable the Gateway API, make sure that the minimum requirements are met before proceeding with the update.
Ensure that your Autopilot cluster version is 1.26 or later and your Standard cluster version is 1.24 or later. To learn more about upgrading your Standard cluster version, see Standard cluster upgrades. To learn more about upgrading your Autopilot cluster version, see Autopilot cluster upgrades.
To enable the Gateway API on an existing GKE cluster (Autopilot or Standard), use the following command. When enabling Gateway on an existing cluster, it might take up to 45 minutes for the cluster to reconcile and install the CRDs.
gcloud container clusters update CLUSTER_NAME \
--location=CLUSTER_LOCATION\
--gateway-api=standard
Replace the following:
CLUSTER_NAME
: the name of the existing cluster.CLUSTER_LOCATION
: the Compute Engine region or zone of the cluster.
The --gateway-api=standard
flag instructs GKE to install the
v1beta1
CRDs with the cluster.
Verify your cluster
After creating or upgrading your cluster, the GKE Gateway controller automatically installs GatewayClasses. It might take a few minutes for the controller to recognize the CRDs and install the GatewayClasses.
Confirm the Gateway API is enabled in the GKE control plane:
gcloud container clusters describe CLUSTER_NAME \ --location=CLUSTER_LOCATION \ --format json
The output is similar to the following. If this output is empty, re-run the cluster update command.
"networkConfig": { ... "gatewayApiConfig": { "channel": "CHANNEL_STANDARD" }, ... },
Confirm the GatewayClasses are installed in your cluster:
kubectl get gatewayclass
The output is similar to the following:
NAME CONTROLLER ACCEPTED AGE gke-l7-global-external-managed networking.gke.io/gateway True 16h gke-l7-regional-external-managed networking.gke.io/gateway True 16h gke-l7-gxlb networking.gke.io/gateway True 16h gke-l7-rilb networking.gke.io/gateway True 16h
To understand the capabilities of each GatewayClass, see GatewayClass capabilities.
Only single-cluster GatewayClasses are installed automatically. To install and use the multi-cluster GatewayClasses for internal and external multi-cluster load balancing, see Enabling multi-cluster Gateways.
Deploy an internal Gateway
An internal Gateway exposes applications that are only reachable from within the VPC or networks connected to the VPC.
Deploy a regional internal Gateway
The following example shows you how to deploy a regional internal Gateway that enables efficient and secure communication between services within a specific geographic region.
Configure a proxy-only subnet
You must configure a proxy-only subnet before you create a Gateway that uses an internal Application Load Balancer. Each region of a VPC in which you use internal Application Load Balancers must have a proxy-only subnet. This subnet provides internal IP addresses to the load balancer proxies.
Create a proxy-only subnet:
gcloud compute networks subnets create SUBNET_NAME \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE \ --region=COMPUTE_REGION \ --network=VPC_NETWORK_NAME \ --range=CIDR_RANGE
Replace the following:
SUBNET_NAME
: the name of the proxy-only subnet.COMPUTE_REGION
: the region of the proxy-only subnet.VPC_NETWORK_NAME
: the name of the VPC network in which you create this proxy-only subnet. Ensure this is the same VPC network where your GKE cluster resides and where you deploy the Gateway. This is important for seamless communication between the load balancer and your backend services.CIDR_RANGE
: the primary IP address range of the subnet. You must use a subnet mask no longer than/26
so that at least 64 IP addresses are available for proxies in the region. The recommended subnet mask is/23
.
Verify your proxy-only subnet:
gcloud compute networks subnets describe SUBNET_NAME \ --region=COMPUTE_REGION
The output is similar to the following:
... gatewayAddress: 10.1.1.1 ipCidrRange: 10.1.1.0/24 kind: compute#subnetwork name: proxy-subnet network: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/global/networks/default privateIpGoogleAccess: false privateIpv6GoogleAccess: DISABLE_GOOGLE_ACCESS purpose: REGIONAL_MANAGED_PROXY region: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION role: ACTIVE selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/regions/REGION/subnetworks/proxy-subnet state: READY
Create a Gateway
A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it is derived from. To learn more about the Gateway resource, see the Gateway resource description or the API specification.
In this case, the administrator of the GKE cluster wants to create a Gateway that can be used by different teams to expose their applications internally. The administrator deploys the Gateway, and application teams deploy their Routes independently and attach them to this Gateway.
Save the following Gateway manifest to a file named
gateway.yaml
:kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: internal-http spec: gatewayClassName: gke-l7-rilb listeners: - name: http protocol: HTTP port: 80
This manifest includes the following fields:
gatewayClassName: gke-l7-rilb
: specifies the GatewayClass that this Gateway is derived from.gke-l7-rilb
corresponds to the internal Application Load Balancer.port: 80
: specifies that the Gateway exposes only port 80 for listening for HTTP traffic.
Deploy the Gateway in your cluster:
kubectl apply -f gateway.yaml
Validate that the Gateway has deployed correctly. It might take a few minutes for it to deploy all of its resources.
kubectl describe gateways.gateway.networking.k8s.io internal-http
The output is similar to the following:
Name: internal-http Namespace: default Spec: Gateway Class Name: gke-l7-rilb Listeners: Allowed Routes: Kinds: Group: gateway.networking.k8s.io Kind: HTTPRoute Namespaces: From: Same Name: http Port: 80 Protocol: HTTP Status: Addresses: Type: IPAddress Value: 192.168.1.14 Conditions: Last Transition Time: 1970-01-01T00:00:00Z Message: Waiting for controller Reason: NotReconciled Status: False Type: Scheduled Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 92s networking.gke.io/gateway test/internal-http Normal UPDATE 45s (x3 over 91s) networking.gke.io/gateway test/internal-http Normal SYNC 45s networking.gke.io/gateway SYNC on test/internal-http was a success
At this point, there is a Gateway deployed in your cluster that has provisioned a load balancer and an IP address. The Gateway has no Routes, however, and so it doesn't know how it should send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404. Next, you deploy an application and Routes, which tell the Gateway how to get to application backends.
Deploy the demo applications
Application teams can deploy their applications and Routes independently from
the deployment of Gateways. In some cases the application team might want to
own the Gateway as well and deploy it themselves as a resource dedicated to
their applications. See Route
binding
for different ownership models of Gateways and Routes. In this example however,
the store team deploys their application and an accompanying HTTPRoute to expose
their app through the internal-http
Gateway created in the previous section.
The HTTPRoute resource has many configurable fields for traffic matching. For an explanation of HTTPRoute's fields, see the API specification.
Deploy the store application (store-v1, store-v2, and store-german deployments) to your cluster:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml
This creates three Deployments and three Services which are named store-v1, store-v2, and store-german.
Validate that the application has deployed successfully:
kubectl get pod
The output is similar to the following after the application is running:
NAME READY STATUS RESTARTS AGE store-german-66dcb75977-5gr2n 1/1 Running 0 38s store-v1-65b47557df-jkjbm 1/1 Running 0 14m store-v2-6856f59f7f-sq889 1/1 Running 0 14m
Validate that the Services have been deployed:
kubectl get service
The output shows a Service for each store Deployment:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE store-german ClusterIP 10.48.3.183 <none> 8080/TCP 4s store-v1 ClusterIP 10.48.2.224 <none> 8080/TCP 5s store-v2 ClusterIP 10.48.4.48 <none> 8080/TCP 5s
Deploy the HTTPRoute
Route resources define protocol-specific rules for mapping traffic from a
Gateway to Kubernetes backends. The HTTPRoute
resource does HTTP and
HTTPS traffic matching and filtering and is supported by all of the gke-l7
GatewayClasses.
In this section, you deploy an HTTPRoute, which programs the Gateway with the routing rules needed to reach your store application.
Save the following HTTPRoute manifest to a file named
store-route.yaml
:kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: store spec: parentRefs: - kind: Gateway name: internal-http hostnames: - "store.example.com" rules: - backendRefs: - name: store-v1 port: 8080 - matches: - headers: - name: env value: canary backendRefs: - name: store-v2 port: 8080 - matches: - path: value: /de backendRefs: - name: store-german port: 8080
Deploy the HTTProute in your cluster:
kubectl apply -f store-route.yaml
The
store
HTTPRoute is bound to theinternal-http
Gateway by using theparentRefs
property. These routing rules are configured on the underlying load balancer as in this diagram:These routing rules process HTTP traffic in the following manner:
- Traffic to
store.example.com/de
goes to the Servicestore-german
. - Traffic to
store.example.com
with the HTTP header"env: canary"
goes to the Servicestore-v2
. - The remaining traffic to
store.example.com
goes to the Servicestore-v1
.
- Traffic to
Verify that the HTTPRoute has been deployed:
kubectl describe httproute store
The output is similar to the following:
Name: store Namespace: default Labels: <none> Annotations: <none> API Version: gateway.networking.k8s.io/v1beta1 Kind: HTTPRoute <...> Spec: Hostnames: store.example.com Parent Refs: Group: gateway.networking.k8s.io Kind: Gateway Name: internal-http Rules: Backend Refs: Group: Kind: Service Name: store-v1 Port: 8080 Weight: 1 Matches: Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-v2 Port: 8080 Weight: 1 Matches: Headers: Name: env Type: Exact Value: canary Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-german Port: 8080 Weight: 1 Matches: Path: Type: PathPrefix Value: /de Status: Parents: Conditions: Last Transition Time: 2022-11-01T04:18:52Z Message: Reason: Accepted Status: True Type: Accepted Last Transition Time: 2022-11-01T04:18:52Z Message: Reason: ReconciliationSucceeded Status: True Type: Reconciled Controller Name: networking.gke.io/gateway Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: internal-http Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 24m sc-gateway-controller default/store Normal SYNC 16m (x4 over 23m) sc-gateway-controller Bind of HTTPRoute "default/store" to ParentRef {Group: gateway.networking.k8s.io", <...>
Verify that the HTTPRoute is bound to the Gateway:
kubectl describe gateway
The output is similar to the following:
Name: internal-http Namespace: default Labels: <none> <...> Status: Addresses: Type: IPAddress Value: 10.128.15.203 Conditions: Last Transition Time: 2022-11-01T03:47:01Z Message: Reason: Scheduled Status: True Type: Scheduled Last Transition Time: 2022-11-01T03:47:01Z Message: Reason: Ready Status: True Type: Ready Listeners: Attached Routes: 1 Conditions: Last Transition Time: 2022-11-01T03:47:01Z Message: Reason: Ready Status: True Type: Ready Name: http Supported Kinds: Group: gateway.networking.k8s.io Kind: HTTPRoute <...>
Send traffic to your application
Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.
Retrieve the IP address from the Gateway so that you can send traffic to your application:
kubectl get gateways.gateway.networking.k8s.io internal-http -o=jsonpath="{.status.addresses[0].value}"
The output is an IP address.
Send traffic to this IP address from shell on a virtual machine (VM) instance with connectivity to the cluster. You can create a VM for this purpose. This is necessary because the Gateway has an internal IP address and is only accessible from within your VPC network. Because the
internal-http
is a regional load balancer, the client shell must be within the same region as the GKE cluster.Make a request to store.example.com:
curl https://store.example.com" --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
Replace
GATEWAY_IP_ADDRESS
with the IP address from the previous step.The output from the demo app shows information about the location where the app is running:
{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "store-v1", "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal", "pod_name": "store-v1-84b47c7f58-pmgmk", "pod_name_emoji": "💇🏼♀️", "project_id": "gateway-demo-243723", "timestamp": "2022-10-25T13:31:17", "zone": "ZONE_NAME" }
Test the path match by going to the German version of the store service at
store.example.com/de
:curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
The output confirms that the request was served by a
store-german
Pod:{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "Gutentag!", "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal", "pod_name": "store-german-5cb6474c55-lq5pl", "pod_name_emoji": "🧞♀", "project_id": "gateway-demo-243723", "timestamp": "2022-10-25T13:35:37", "zone": "ZONE_NAME" }
Finally, use the
env: canary
HTTP header to send traffic to the canary version of the store Service:curl -H "env: canary" https://store.example.com" --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
The output confirms that the request was served by a
store-v2
Pod:{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "store-v2", "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal", "pod_name": "store-v2-5788476cbd-s9thb", "pod_name_emoji": "🦰", "project_id": "gateway-demo-243723", "timestamp": "2022-10-25T13:38:26", "zone": "ZONE_NAME" }
Deploy an external Gateway
An external Gateway exposes applications that are reachable from the internet or networks outside of your VPC. The deployment is similar to an internal Gateway deployment except you must secure your applications because the Gateway is accessible to the public internet.
You have two options to create an external Gateway: a global external Gateway or a regional external Gateway.
A global external Gateway uses a Global IP address (or Anycast IP address) as the frontend of the Gateway advertised in all Google Cloud Compute regions. Clients sending traffic to this Anycast IP address are routed to the closest Google location where the IP is advertised. The global external Gateway is only available in the Premium Network Service tier.
A regional external Gateway uses a Regional IP as the frontend of the Gateway advertised only in the local Google Cloud Compute region where the regional external Gateway is deployed. Clients sending traffic to this regional IP address are routed by their local ISP and over the Internet before reaching the Google region where the IP is advertised. The regional external Gateway is only available in the Standard Network Service tier.
Deploy a global external Gateway
The following example shows you how to expose a store application with multiple certificates attached to the global external Gateway and grouped in a certificate map using Certificate Manager and an HTTPRoute.
Create a certificate map
Google recommends that you use Certificate Manager to manage certificates when you need 15 or more certificates per Gateway or you need to use wildcard certificates.
You can also secure your external Gateway using Kubernetes Secrets or Google-managed SSL certificates. For more information, see Gateway security.
In this section, you create certificates using Certificate Manager to secure the applications running on the cluster.
Enable the Certificate Manager API:
gcloud services enable certificatemanager.googleapis.com
Create a certificate map:
gcloud beta certificate-manager maps create store-example-com-map
Load your Google-managed certificate and keys into a Certificate:
gcloud beta certificate-manager certificates create store-example-com-cert \ --certificate-file="CERTIFICATE_FILE" \ --private-key-file="PRIVATE_KEY_FILE"
Replace the following:
CERTIFICATE_FILE
: the name of the new file that you choose. The file must have the extension.pem
. For example,cert.pem
.PRIVATE_KEY_FILE
: the name of your private key file.
For more information, see Create a private key and certificate.
Create a
CertificateMapEntry
which assigns the certificate to the certificate map:gcloud beta certificate-manager maps entries create store-example-com-map-entry \ --map=store-example-com-map \ --hostname=store.example.com \ --certificates=store-example-com-cert
To learn how to secure a Gateway using other sources for certificates, such as Kubernetes Secrets or SSL certificates, see Secure a Gateway.
Create a Gateway
A Gateway resource represents a data plane that routes traffic in Kubernetes. A Gateway can represent many different kinds of load balancing and routing depending on the GatewayClass it uses.
To learn more about the Gateway resource, see the Gateway resource description or the API specification.
In this section, you create a Gateway. Application teams can use the Gateway to expose their applications to the internet by deploying Routes independently and attaching them securely to the Gateway.
Save the following manifest to a file named
gateway.yaml
:kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: external-http annotations: networking.gke.io/certmap: store-example-com-map spec: gatewayClassName: gke-l7-global-external-managed listeners: - name: https protocol: HTTPS port: 443
This manifest describes a Gateway with the following fields:
gatewayClassName: gke-l7-global-external-managed
: specifies the GatewayClass for this Gateway. This gateway class uses a global external Application Load Balancer.protocol: HTTPS
andport: 443
: specify that the Gateway exposes port 443 for HTTPS traffic. These fields enables TLS.networking.gke.io/certmap: store-example-com-map
: specifies the name of the certificate map in Certificate Manager.
There is no TLS section because TLS is configured with Certificate Manager using the annotation
networking.gke.io/certmap
.Apply the manifest to your cluster:
kubectl apply -f gateway.yaml
It might take a few minutes for GKE to deploy the resources.
Verify that the Gateway has deployed successfully:
kubectl describe gateway
The output is similar to the following:
Name: external-http Namespace: default Labels: <none> ... Spec: Gateway Class Name: gke-l7-global-external-managed Listeners: Allowed Routes: Namespaces: From: Same Name: https Port: 443 Protocol: HTTPS Tls: Certificate Refs: Group: Kind: Secret Name: store-example-com Mode: Terminate ...
This output shows that the Gateway deployed in your cluster has a load balancer and a public IP address. The Gateway has no Routes, which means it can't send traffic to backends. Without Routes, all traffic goes to a default backend, which returns an HTTP 404 response. In the next section, you deploy Routes, which instruct the Gateway to send traffic to backends.
Deploy the demo applications
Application teams can deploy their applications and Routes independently from
the deployment of Gateways. In some cases the application team might want to own
the Gateway as well and deploy it themselves as a resource dedicated to their
applications. See
Route binding
for different ownership models of Gateways and Routes. In this example, the
store team deploys their application and an accompanying HTTPRoute to expose
their app through the external-http
Gateway created in the previous section.
For more information about HTTPRoute fields, see the API specification.
Deploy the sample application to your cluster:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/store.yaml
This sample application creates three Deployments and three Services which are named
store-v1
,store-v2
, andstore-german
.Verify that the application has deployed successfully:
kubectl get pod
The output is similar to the following:
NAME READY STATUS RESTARTS AGE store-german-66dcb75977-5gr2n 1/1 Running 0 38s store-v1-65b47557df-jkjbm 1/1 Running 0 14m store-v2-6856f59f7f-sq889 1/1 Running 0 14m
Verify that the Services have deployed successfully:
kubectl get service
The output is similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE store-german ClusterIP 10.48.3.183 <none> 8080/TCP 4s store-v1 ClusterIP 10.48.2.224 <none> 8080/TCP 5s store-v2 ClusterIP 10.48.4.48 <none> 8080/TCP 5s
Create an HTTPRoute
Route resources define protocol-specific rules for mapping traffic from a
Gateway to Kubernetes backends. The
HTTPRoute resource
does HTTP and HTTPS traffic matching and filtering and is supported by all of
the gke-l7-*
GatewayClasses.
In this section, you deploy an HTTPRoute, which configures the Gateway with routing rules required to reach the sample application.
Save the following manifest to a file named
store-route-external.yaml
:kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: store-external spec: parentRefs: - kind: Gateway name: external-http hostnames: - "store.example.com" rules: - backendRefs: - name: store-v1 port: 8080 - matches: - headers: - name: env value: canary backendRefs: - name: store-v2 port: 8080 - matches: - path: value: /de backendRefs: - name: store-german port: 8080
This manifest describes an HTTPRoute that references the
external-http
Gateway.Apply the manifest to your cluster:
kubectl apply -f store-route-external.yaml
The
store
HTTPRoute is bound to theexternal-http
Gateway by using theparentRefs
property. These following diagram shows the routing rules configured on the underlying load balancer:The routing rules process HTTP traffic as follows:
- Traffic to
store.example.com/de
routes to Servicestore-german
. - Traffic to
store.example.com
with the HTTP header"env: canary"
routes to Servicestore-v2
. - The remaining traffic to
store.example.com
routes to Servicestore-v1
.
- Traffic to
Verify that the HTTPRoute has been deployed:
kubectl describe httproute store-external
The output is similar to the following:
Name: store-external Namespace: default Labels: <none> Annotations: <none> API Version: gateway.networking.k8s.io/v1beta1 Kind: HTTPRoute <...> Spec: Hostnames: store.example.com Parent Refs: Group: gateway.networking.k8s.io Kind: Gateway Name: external-http Rules: Backend Refs: Group: Kind: Service Name: store-v1 Port: 8080 Weight: 1 Matches: Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-v2 Port: 8080 Weight: 1 Matches: Headers: Name: env Type: Exact Value: canary Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-german Port: 8080 Weight: 1 Matches: Path: Type: PathPrefix Value: /de Status: Parents: Conditions: Last Transition Time: 2022-11-01T05:42:31Z Message: Reason: Accepted Status: True Type: Accepted Last Transition Time: 2022-11-01T05:43:18Z Message: Reason: ReconciliationSucceeded Status: True Type: Reconciled Controller Name: networking.gke.io/gateway Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: external-http Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 2m48s sc-gateway-controller default/store-external Normal SYNC 61s (x3 over 2m27s) sc-gateway-controller Bind of HTTPRoute "default/store-external" to ParentRef Group: "gateway.networking.k8s.io", ...
Verify that the HTTPRoute is bound to the Gateway:
kubectl describe gateway external-http
The output is similar to the following:
Name: external-http Namespace: default Labels: <none> <...> Status: Addresses: Type: IPAddress Value: 34.149.207.45 Conditions: Last Transition Time: 2022-11-01T05:37:21Z Message: Reason: Scheduled Status: True Type: Scheduled Last Transition Time: 2022-11-01T05:43:18Z Message: Reason: Ready Status: True Type: Ready Listeners: Attached Routes: 1 Conditions: Last Transition Time: 2022-11-01T05:43:18Z Message: Reason: Ready Status: True Type: Ready Name: https Supported Kinds: Group: gateway.networking.k8s.io Kind: HTTPRoute <...>
Send traffic to your application
Now that your Gateway, Route, and application are deployed in your cluster, you can pass traffic to your application.
Get the IP address of the Gateway:
kubectl get gateways.gateway.networking.k8s.io external-http -o=jsonpath="{.status.addresses[0].value}"
The output is an IP address.
Create a VM:
gcloud cloud-shell ssh
Send traffic to the Gateway IP address from the VM. You must set the host header manually because you do not own the
example.com
hostname.curl https://store.example.com --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
Replace
GATEWAY_IP_ADDRESS
with the IP address of the Gateway from the previous step.The output shows information from the demo app about the location where the app is running:
{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "store-v1", "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal", "pod_name": "store-v1-84b47c7f58-pmgmk", "pod_name_emoji": "💇🏼♀️", "project_id": "gateway-demo-243723", "timestamp": "2022-09-25T13:31:17", "zone": "us-central1-a" }
Test the path match by going to the German version of the
store
service atstore.example.com/de
:curl https://store.example.com/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
The output confirms that the request was served by a
store-german
Pod:{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "Gutentag!", "node_name": "gke-gke1-pool-2-bd121936-n3xn.c.gateway-demo-243723.internal", "pod_name": "store-german-5cb6474c55-lq5pl", "pod_name_emoji": "🧞♀", "project_id": "gateway-demo-243723", "timestamp": "2022-09-25T13:35:37", "zone": "us-central1-a" }
Send traffic to the canary version of the
store
Service using theenv: canary
HTTP header:curl -H "env: canary" https://store.example.com"/de --resolve store.example.com:443:GATEWAY_IP_ADDRESS --cacert cacert.pem -v
The output confirms that the request was served by a
store-v2
Pod:{ "cluster_name": "gke1", "host_header": "store.example.com", "metadata": "store-v2", "node_name": "gke-gke1-pool-2-bd121936-5pfc.c.gateway-demo-243723.internal", "pod_name": "store-v2-5788476cbd-s9thb", "pod_name_emoji": "👩🏿", "project_id": "gateway-demo-243723", "timestamp": "2022-09-25T13:38:26", "zone": "us-central1-a" }
Deploy a regional external Gateway
The following example shows you how to expose a store application with multiple certificates attached to the regional external Gateway using self-managed certificates and an HTTPRoute.
Create a proxy subnet for your regional Gateway
You must configure a proxy-only subnet
before you create a Gateway that uses a regional external Application Load Balancer. Each region of
a VPC in which you use regional external Application Load Balancer must have an
external_managed_proxy
subnet. This subnet provides internal IP
addresses to the load balancer proxies.
Create a certificate to secure your client traffic
You can use a certificate issued and validated by your certificate authority (CA) or create a self-signed certificate. For more information on how to create a certificate, see Store a certificate in a Kubernetes Secret.
CertificateMap or Google-managed SSL certificates are not supported with regional Gateways. Use self-managed regional SSL certificates or secrets to secure traffic between your clients and your regional Gateway. For more information about Certificate and Google Cloud load balancers, see Certificates and Google Cloud load balancers
Create a regional external HTTP(S) Gateway
Create a regional static IP address for the external load balancer.
gcloud compute addresses create IP_ADDRESS_NAME \ --region=COMPUTE_REGION \ --network-tier=STANDARD
Replace the following:
IP_ADDRESS_NAME
: the name of the new static IP address.COMPUTE_REGION
: The Compute Engine region where your cluster is running.
Create a regional external Application Load Balancer Gateway using a self-managed certificate as follows and save the manifest as
regional-gateway.yaml
:kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: external-regional-http spec: gatewayClassName: gke-l7-regional-external-managed listeners: - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - name: store-example-com addresses: - type: NamedAddress value: IP_ADDRESS_NAME
Apply the
regional-gateway
manifest:kubectl apply -f regional-gateway.yaml
Verify your configuration.
kubectl get gateway
The output is similar to the following:
NAME CLASS ADDRESS READY AGE external-http gke-l7-regional-external-managed 35.118.32.224 True 49s
To get more details, use a describe command:
kubectl describe gateway
The output is similar to the following:
Name: external-regional-http Namespace: default Labels: <none> ... Spec: Gateway Class Name: gke-l7-regional-external-managed Listeners: Allowed Routes: Namespaces: From: Same Name: https Port: 443 Protocol: HTTPS Tls: Certificate Refs: Group: Kind: Secret Name: store-example-com Mode: Terminate ...
Deploy the demo application
You can deploy your applications and routes independently from the deployment of Gateways.
For more information on how to deploy the demo applications, see Deploy the demo applications.Create an HTTPRoute
You must create an HTTPRoute to do HTTP and HTTPS traffic matching and filtering.
Send Traffic to your application
After you have deployed your application and created HTTPRoutes, you can pass traffic to your application.
For more information on how to send traffic to your application, see Send traffic to your application.Use shared Gateways
The Gateway APIs use separate resources, Gateways and Route resources, to deploy load balancers and routing rules. This differs from Ingress, which combines everything in one resource. By splitting responsibility among resources, Gateway enables the load balancer and its routing rules to be deployed separately and to be deployed by different users or teams. This enables Gateways to become shared Gateways that attach with many different Routes that can be fully owned and managed by independent teams, even across different namespaces.
Deploy routes against a shared Gateway
This example builds on the internal-http
Gateway deployed in
Deploy an internal Gateway.
In this example, the site team deploys their application, Services, and an HTTPRoute to match traffic from the Gateway to those Services.
Deploy the example application:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/app/site.yaml
Save the following manifest to a file named
site-route-internal.yaml
:kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: site-internal spec: parentRefs: - kind: Gateway name: internal-http hostnames: - "site.example.com" rules: - backendRefs: - name: site-v1 port: 8080
This manifest describes an HTTPRoute that matches all traffic for
site.example.com
and routes it to thesite-v1
Service.Apply the manifest to your cluster:
kubectl apply -f site-route-internal.yaml
Verify that the HTTPRoute is attached to the Gateway:
kubectl describe httproute.gateway.networking.k8s.io site-internal
The output is similar to the following:
Status: Parents: Conditions: Last Transition Time: 2023-01-09T15:05:43Z Message: Reason: Accepted Status: True Type: Accepted Last Transition Time: 2023-01-09T15:05:43Z Message: Reason: ReconciliationSucceeded Status: True Type: Reconciled Controller Name: networking.gke.io/gateway Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: internal-http ...
If the
Accepted
condition for the Gateway isTrue
, the HTTPRoute has successfully bound to the Gateway. To learn more about the Status field, see route status.Verify that traffic to the Gateway is routed correctly:
curl -H "host: site.example.com" GATEWAY_IP_ADDRESS curl -H "host: store.example.com" GATEWAY_IP_ADDRESS
Replace
GATEWAY_IP_ADDRESS
with the IP address of the Gateway.You must use a virtual machine (VM) in the same VPC as the Gateway.
The output is similar to the following:
{ "cluster_name": "CLUSTER_NAME", "host_header": "site.example.com", "metadata": "site-v1", "pod_name": "site-v1-5d64fc4d7d-fz6f6", "pod_name_emoji": "👩🏼🍳", "project_id": "PROJECT_ID", "timestamp": "2022-11-02T19:07:01", "zone": "ZONE_NAME" } ... { "cluster_name": "CLUSTER_NAME", "host_header": "store.example.com", "metadata": "store-v1", "pod_name": "store-v1-6d8d58d78-vz8pn", "pod_name_emoji": "🧝🏻♂️", "project_id": "PROJECT_ID", "timestamp": "2022-11-02T19:07:01", "zone": "ZONE_NAME" }
Configure the Gateway default backend
All of the gke-l7-*
GatewayClasses return HTTP 404 to unmatched traffic. You
can configure the default backend using an explicit default Route that sends
unmatched traffic to a user-provided Service.
Gateways are configured to handle error codes like 404 (Not Found) and 500 (Server Error), even without explicit backend definitions. The default behavior may vary between Gateway implementations. For greater control over error handling, consider configuring custom backends.
The following HTTPRoute is an example of how to customize the default backend. If you apply an HTTPRoute similar to the following, it takes precedence over the implicit default backend:
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: custom-default-backend
spec:
parentRefs:
- kind: Gateway
name: my-internal-gateway
rules:
- backendRefs:
- name: my-custom-default-backend-service
port: 8080
This HTTPRoute matches all traffic from a particular Gateway. You can only have one such rule for each Gateway or else the rules conflict and precedence ordering applies.
You can use a default backend to prevent someone from creating a default route Backend that routes all Gateway traffic. An explicit HTTPRoute always takes precedence over new HTTPRoutes with conflicting routing rules.
Configure a static IP address for a Gateway
Every Gateway has an IP address it uses to listen for traffic. If you do not specify an IP address on the Gateway, then the Gateway controller automatically provides an IP address. You can also create a static IP address so that the IP address exists independent of the Gateway lifecycle.
After a Gateway is deployed, its IP address shows in the status field:
kind: Gateway
...
status:
addresses:
- value: 10.15.32.3
Depending on the GatewayClass, the IP address is allocated from the following subnets:
GatewayClasses | Default IP Address Pool |
---|---|
|
Regional private IP addresses from the primary node IPv4/IPv6 address range |
|
Regional public IP addresses from Google's regional external IPv4/IPv6 ranges |
|
Global public IP addresses from Google's global external IPv4/IPv6 ranges |
The field addresses.NamedAddress
lets you specify an IP address independently
of the Gateway. You can create a static IP address resource prior to Gateway
deployment and the resource is referenced by the NamedAddress
. You can reuse
the static IP address even if the Gateway is deleted.
Use a named IP address
You can configure an IPv4 or IPv6 address by specifying a NamedAddress
. You
must provision a static IP address before you create a Gateway.
Create a static IP address resource:
gcloud compute addresses create IP_ADDRESS_NAME \ --purpose=SHARED_LOADBALANCER_VIP \ --region=COMPUTE_REGION \ --subnet=SUBNET \ --project=PROJECT_ID
Replace the following:
IP_ADDRESS_NAME
: the name of the new static IP addressCOMPUTE_REGION
: for regional Gateways, the Compute Engine region where your cluster is running. This flag is not needed for global, external Gateways.SUBNET
: the subnet for the IP address. This flag is not needed for global, external Gateways.PROJECT_ID
: the project where your GKE cluster is running.
Save the following manifest to a file named
named-ip-gateway.yaml
:kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: internal-http spec: gatewayClassName: gke-l7-rilb listeners: - name: http protocol: HTTP port: 80 addresses: - type: NamedAddress value: IP_ADDRESS_NAME
This manifest describes a Gateway that references the named IP address.
Apply the manifest to your cluster:
kubectl apply -f named-ip-gateway.yaml
Verify your Gateway IP address:
kubectl describe gateway internal-http
The output is similar to the following:
Name: internal-http Namespace: default Labels: <none> ... Spec: Addresses: Type: NamedAddress Value: IP_ADDRESS_NAME Gateway Class Name: gke-l7-rilb Listeners: Allowed Routes: Namespaces: From: Same Name: http Port: 80 Protocol: HTTP Status: Addresses: Type: IPAddress Value: 10.15.32.103
Configure HTTP-to-HTTPS redirects
Cloud Load Balancing offers HTTP to HTTPS redirect functionality. An external Application Load Balancer redirects unencrypted HTTP requests to an HTTPS load balancer that uses the same IP address. When you create a Gateway with HTTP-to-HTTPS redirects enabled, both of these load balancers are created automatically. Requests to the external IP address of the Gateway on port 80 are automatically redirected to the same external IP address on port 443.
By default, HTTP to HTTPS redirects are not defined on the Gateway.
To redirect HTTP traffic to HTTPS, configure a Gateway to handle both HTTP and HTTPS traffic. If you disable either HTTP or HTTPS, the Gateway does not redirect traffic.
The following example shows you how you can use HTTP-to-HTTPS redirect as a means to ensure that traffic from your clients going to your web applications is always being redirected to a secure page.
HTTP-to-HTTPS redirects are not supported with the gke-l7-gxlb
and
gke-l7-gxlb-mc
GatewayClasses. To learn more about the different features
supported with each GatewayClass, see
GatewayClass capabilities.
Redirect HTTP traffic from an infrastructure namespace
In some cases, there isn't a clear distinction between the infrastructure or platform admin team and the application teams and preventing misuse of the Gateway can become a challenge.
The following example further restricts the use of the HTTP listener to prevent unintentional use of non-secure protocol from the application teams. This example configures the Gateway to allow an HTTPRoute to use the HTTP listener only if the route is in a given namespace (http-redirect) while it opens the HTTPS listener to all namespaces. You can restrict the http-redirect namespace using Kubernetes RBAC so that application teams cannot create an HTTPRoute in this namespace by mistake.
Create the namespace of a Gateway. Save the manifest as
gateway-namespace.yaml
:apiVersion: v1 kind: Namespace metadata: name: gateway-infra
Apply the manifest:
kubectl apply -f gateway-namespace.yaml
Create the namespace of a Gateway and save the manifest as
redirect-namespace.yaml
:apiVersion: v1 kind: Namespace metadata: name: http-redirect labels: otherInfra: httpToHttps
- Specific labels are set for this namespace.
Apply the manifest:
kubectl apply -f redirect-namespace.yaml
To restrict the http listener usage, create a Gateway using the following manifest. Save the manifest as
external-gateway.yaml
:kind: Gateway apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: external-http namespace: gateway-infra spec: gatewayClassName: gke-l7-global-external-managed listeners: - name: http protocol: HTTP port: 80 allowedRoutes: kinds: - kind: HTTPRoute namespaces: from: selector selector: matchLabels: otherInfra: httpToHttps - name: https protocol: HTTPS port: 443 allowedRoutes: kinds: - kind: HTTPRoute namespaces: from: All tls: mode: Terminate options: networking.gke.io/pre-shared-certs: store-example-com ```
The
namespace
field specifies that the Gateway is created in thegateway-infra
namespace.The
namespaces
field in theallowedRoutes
section restricts the http listener to the namespace matching the labelotherInfra: httpToHttps
.
Apply the manifest:
kubectl apply -f external-gateway.yaml
To force the HTTPS redirect, create a default HTTPRoute using the following manifest. Save the manifest as
http-redirect.yaml
:kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: redirect namespace: http-redirect spec: parentRefs: - namespace: gateway-infra name: external-http sectionName: http rules: - filters: - type: RequestRedirect requestRedirect: scheme: https
- The
sectionName
field instructs the Gateway to match only on the http listener. TheRequestRedirect
filter forces the redirection to the https listener.
- The
Apply the manifest:
kubectl apply -f http-redirect.yaml
Create a Service for an application using the following manifest. Save the manifest as
service-deployment.yaml
:apiVersion: v1 kind: Service metadata: name: store-v1 spec: selector: app: store version: v1 ports: - port: 8080 targetPort: 8080 --- apiVersion: apps/v1 kind: Deployment metadata: name: store-v1 spec: replicas: 2 selector: matchLabels: app: store version: v1 template: metadata: labels: app: store version: v1 spec: containers: - name: whereami image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.20 ports: - containerPort: 8080 env: - name: METADATA value: "store-v1"
Apply the manifest:
kubectl apply -f service-deployment.yaml
Create an HTTPRoute for an application that only allows HTTPS using the following manifest. Save the manifest as
http-route.yaml
:kind: HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 metadata: name: store-external labels: gateway: external-http spec: parentRefs: - name: external-http namespace: gateway-infra sectionName: https hostnames: - "store.example.com" rules: - backendRefs: - name: store-v1 port: 8080
Apply the manifest:
kubectl apply -f http-route.yaml
Configure path redirects and URL rewrites
Path redirects involve redirecting an incoming request from one URL path to another. Path redirects let you change the structure of the URL when you need to handle outdated or deprecated URLs.
URL rewrites help modify the incoming URL before processing it on the server. It allows you to change the structure or format of the URL without actually changing the underlying content or file structure. URL rewriting is beneficial for creating user-friendly and SEO-friendly URLs that are easy to remember and understand. By default, path redirects and URL rewrites are not configured, you need to explicitly configure those redirects or rewrites using a filter in your HTTPRoute.
GKE Gateway supports path redirects and URL rewrites. For more information, see HTTP path redirects and rewrites.
Configure path redirects
You can configure path redirects to either replace the entire path or only a prefix in the URL.
Replace entire path
To replace an entire path, configure a filter in an HTTPRoute that replaces any URL that contains the prefix
/any-path
in the URL path by the strict value/new-path
.Create an
HTTPRoute
manifest as follows and name it asstore.yaml
:apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: store spec: parentRefs: - kind: Gateway name: external-http hostnames: - store.example.com rules: - matches: - path: type: PathPrefix value: /any-path filters: - type: RequestRedirect requestRedirect: path: type: ReplaceFullPath replaceFullPath: /new-path statusCode: 302
For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/... should be redirected to a new location, https://store.example.com/new-path/ (strict).
Apply the manifest:
kubectl apply -f store.yaml
This routing rule follows a strict redirection rule, which means that the browser does not attempt to cache the redirect, instead, redirects to the latest version.
Replace a prefix only
To replace a prefix only, configure a filter in an HTTPRoute that replaces any URL that contains the prefix
/any-prefix
in the URL path by the strict value/new-prefix
.Create an
HTTPRoute
manifest as follows and name it asstore.yaml
:apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: store spec: parentRefs: - kind: Gateway name: external-http hostnames: - store.example.com rules: - matches: - path: type: PathPrefix value: /any-prefix filters: - type: RequestRedirect requestRedirect: path: type: ReplacePrefixMatch replacePrefixMatch: /new-prefix statusCode: 302
For example, this manifest sets a routing rule for an HTTPRoute as follows: Any route to the URL https://store.example.com/any-path/v1/... should be redirected to a new location, https://store.example.com/new-path/v1/... (only).
Apply the manifest:
kubectl apply -f store.yaml
This routing rule follows the only redirection rule, which ensures that the browser always redirects you to the same intended page.
Configure URL rewrites
Set URL rewrites to change the way a URL appears to users. You can use URL rewrites to make URLs more user-friendly, to improve SEO, or to redirect users to a new page.
Rewrite the entire hostname
To rewrite the entire hostname:
Configure a filter in an HTTPRoute that instructs the Gateway to replace the
Host
information in the request header from www.example.com to store.example.com before forwarding the request to the backend service.Create an
HTTPRoute
manifest as follows and name it aswww.yaml
:apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: www spec: parentRefs: - kind: Gateway name: external-http hostnames: - www.example.com rules: - filters: - type: URLRewrite urlRewrite: hostname: store.example.com backendRefs: - name: store-v1 port: 8080
For example, with the above configuration, any request to https://www.example.com is forwarded to the backend service with the
Host: store.example.com
header, instead ofHost: www.example.com
.Apply the manifest:
kubectl apply -f www.yaml
Rewrite using path modifiers
You can combine rewrites with path modifiers to provide advanced URL and path modifications before relaying the request to the backend service.
To rewrite using path modifiers:
Configure a filter in an HTTPRoute that instructs the Gateway to replace the 'Host' information in the request header from www.example.com to store.example.com and replace the value
/store
by/
before forwarding the request to the backend service.Create an
HTTPRoute
manifest as follows and name it aswww.yaml
:apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: www spec: parentRefs: - kind: Gateway name: external-http hostnames: - www.example.com rules: - matches: - path: type: PathPrefix value: /store filters: - type: URLRewrite urlRewrite: hostname: store.example.com path: type: ReplacePrefixMatch replacePrefixMatch: /de backendRefs: - name: store-german port: 8080
For example, with the above configuration, any request to https://www.example.com/store/... is forwarded to the backend service with
Host: store.example.com
in the request header (instead ofHost: www.example.com
) and the/store
is rewritten to/de
.Apply the manifest:
kubectl apply -f www.yaml
Verify your configuration
To verify that the filter was applied after creating your HTTPRoute with URL rewrite or path redirects filters, do the following:
kubectl get httproute www -o yaml
The output is similar to the following:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1beta1","kind":"HTTPRoute","metadata":{"annotations":{},"name":"www","namespace":"default"},"spec":{"hostnames":["www.example.com"],"parentRefs":[{"kind":"Gateway","name":"external-http"}],"rules":[{"backendRefs":[{"name":"store-german","port":8080}],"filters":[{"type":"URLRewrite","urlRewrite":{"hostname":"store.example.com","path":{"replacePrefixMatch":"/de","type":"ReplacePrefixMatch"}}}],"matches":[{"path":{"type":"PathPrefix","value":"/store"}}]}]}}
creationTimestamp: "2023-06-22T01:00:42Z"
generation: 3
name: www
namespace: default
resourceVersion: "51268631"
uid: e516493e-806d-44d6-ae0d-1c9ff25682cf
spec:
hostnames:
- www.example.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: external-http
rules:
- backendRefs:
- group: ""
kind: Service
name: store-german
port: 8080
weight: 1
filters:
- type: URLRewrite
urlRewrite:
hostname: store.example.com
path:
replacePrefixMatch: /de
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /store
status:
parents:
- conditions:
- lastTransitionTime: "2023-06-22T01:11:26Z"
message: ""
observedGeneration: 2
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2023-06-22T01:11:26Z"
message: ""
observedGeneration: 2
reason: ReconciliationSucceeded
status: "True"
type: Reconciled
controllerName: networking.gke.io/gateway
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: external-http
To get more details, use the describe command:
kubectl describe httproute
Configure custom request and response headers
Custom request and response headers let you specify additional headers to HTTP(S) requests and responses. Depending on the information detected by the load balancer, these headers can include the following information:
- Latency to the client
- Geographic location of the client's IP address
- Parameters of the TLS connection
By default, there are no custom headers added to the request sent/received to/from your backend services, you need to explicitly configure custom headers using a filter in your HTTPRoute.
You can configure custom headers by adding a filter section in your HTTPRoute's rules as follows:
Configure custom request headers
Create a HTTPRoute manifest with a RequestHeaderModifier filter and save it as
http-route-request.yaml:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: store
spec:
<...>
rules:
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
<...>
Apply the manifest:
kubectl apply -f http-route-request.yaml
Configure custom response headers
Create a HTTPRoute manifest with a ResponseHeaderModifier filter and save it as
http-route-response.yaml:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: store
spec:
<...>
rules:
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
<...>
Apply the manifest:
kubectl apply -f http-route-response.yaml
You can add, set, and remove headers as described in the Gateway API implementation. You can configure your HTTPRoute with a custom header using Google Cloud supported variables.
Example 1:
To configure an HTTPRoute that adds client location information to the HTTP
request before sending it to the backend service, create a HTTPRoute manifest
and name it as external-http-request.yaml
:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: store
spec:
parentRefs:
- kind: Gateway
name: external-http
hostnames:
- store.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /fr
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: X-Client-Geo-Location
value: "{client_region},{client_city}"
backendRefs:
- name: store-french
port: 8080
For example, for clients located in Strasbourg, France, the Gateway adds a
header as X-Client-Geo-Location:FR,Strasbourg
.
Example 2:
To configure an HTTPRoute that adds a custom response header to support HTTP
Strict Transport Security, create a HTTPRoute manifest and name it as
external-http-response.yaml
:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: store
spec:
parentRefs:
- kind: Gateway
name: external-http
hostnames:
- store.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /de
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
add:
- name: Strict-Transport-Security
value: max-age=63072000
backendRefs:
- name: store-german
port: 8080
Verify your configuration
To verify your configuration after configuring custom request and response headers, do the following:
kubectl get httproute
The output is similar to the following:
NAME HOSTNAMES AGE store ["store.example.com"] 4d23h
To get more details, use the describe command:
kubectl describe httproute
The output is similar to the following:
Name: store Namespace: default Labels: <none> Annotations: <none> API Version: gateway.networking.k8s.io/v1beta1 Kind: HTTPRoute Metadata: Creation Timestamp: 2023-05-27T00:51:01Z Generation: 5 Resource Version: 25418887 UID: 2e07a1b8-420b-41b4-acd1-cecbfcd39f42 Spec: Hostnames: store.example.com Parent Refs: Group: gateway.networking.k8s.io Kind: Gateway Name: external-http Rules: Backend Refs: Group: Kind: Service Name: store-v1 Port: 8080 Weight: 1 Matches: Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-v2 Port: 8080 Weight: 1 Matches: Headers: Name: env Type: Exact Value: canary Path: Type: PathPrefix Value: / Backend Refs: Group: Kind: Service Name: store-german Port: 8080 Weight: 1 Filters: Request Header Modifier: Add: Name: X-Client-Geo-Location Value: {client_region},{client_city} Type: RequestHeaderModifier Matches: Path: Type: PathPrefix Value: /de Status: <...>
Route status
HTTPRoute resources emit conditions and events to help users understand if a HTTPRoute has successfully bound with one or more Gateways or if it was rejected.
HTTPRoute conditions
HTTPRoute conditions indicate the status of the Route and the Gateways it is bound to. Because a Route can be bound to multiple Gateways, this is a list of Gateways and the individual conditions between the Route and each Gateway.
Accepted=True
indicates that the HTTPRoute is successfully bound to a Gateway.Accepted=False
indicates that the HTTPRoute has been rejected from binding with this Gateway.
If there are no Gateways listed under the Gateway bindings
heading, then your
HTTPRoute labels and Gateway label selectors might not match. This can occur if
your Route is not being selected by any Gateways.
HTTPRoute events
HTTPRoute events provide details about the status of the HTTPRoute. Events are grouped by the following reasons:
ADD
events are triggered by a resource being added.UPDATE
events are triggered by a resource being updated.SYNC
events are triggered by periodic reconciliation.
Route merging, precedence, and validation
Route precedence
The Gateway API defines strict precedence rules for how traffic is matched by Routes that have overlapping routing rules. The precedence between two overlapping HTTPRoutes is as follows:
- Hostname merge: The longest/most specific hostname match.
- Path merge: The longest/most specific path match.
- Header merge: The largest number of HTTP headers that match.
- Conflict: If the previous three rules don't establish precedence, then precedence goes to the HTTPRoute resource with the oldest timestamp.
Route merging
For gke-l7
GatewayClasses, all HTTPRoutes for a given Gateway are merged into
the same URL map resource. How the HTTPRoutes
are merged together depends on the type of overlap between HTTPRoutes. The
HTTPRoute from the earlier example can be split into three separate
HTTPRoutes to illustrate route merging and precedence:
- Route merge: All three HTTPRoutes attach with the same
internal-http
Gateway, so they are merged together. - Hostname merge: All three Routes match for
store.example.com
, so their hostname rules are merged. - Path merge: store-german-route has a more specific path
/de
, so this is not merged further. store-v1-route and store-v2-route both match on the same/*
path as well, so they are merged on the path. - Header merge: store-v2-route has a more specific set of HTTP header matches than store-v1-route, so they are not merged further.
- Conflict: Because the Routes are able to be merged on hostname, path, and headers, there are no conflicts, and all of the routing rules are applied to traffic.
The single HTTPRoute used in the earlier example are equivalent to these three separate routes:
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: store-v1-route
spec:
parentRefs:
- kind: Gateway
name: internal-http
hostnames:
- "store.example.com"
rules:
- backendRefs:
- kind: Service
name: store-v1
port: 8080
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: store-v2-route
spec:
parentRefs:
- kind: Gateway
name: internal-http
hostnames:
- "store.example.com"
rules:
- matches:
- headers:
- type: Exact
name: env
value: canary
backendRefs:
- kind: Service
name: store-v2
port: 8080
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: store-german-route
spec:
parentRefs:
- kind: Gateway
name: internal-http
hostnames:
- "store.example.com"
rules:
- matches:
- path:
type: PathPrefix
value: /de
backendRefs:
- kind: Service
name: store-german
port: 8080
Kubernetes Gateways and Istio Gateways
Note that the Kubernetes Gateway API and the Istio API both have a resource
named Gateway
. While they perform similar functions, they are not the same
resource. If you are using Istio and the Gateway API in the same Kubernetes
cluster, these names overlap when using kubectl on the command line.
kubectl get gateway
might return the Kubernetes Gateway resources and not the
Istio Gateway resources or vice versa.
$ kubectl api-resources
NAME SHORTNAMES APIGROUP NAMESPACED KIND
gateways gw networking.istio.io/v1beta1 true Gateway
gateways gtw networking.k8s.io/v1beta1 true Gateway
If you are using Istio and upgrade to GKE 1.20 and later it is
recommended to start using the Gateway resource shortname or specify the API
group. The shortname for a Kubernetes Gateway is gtw
and the shortname for an
Istio Gateway is gw
. The following commands return the Kubernetes Gateway and
Istio Gateway resources respectively.
# Kubernetes Gateway
$ kubectl get gtw
NAME CLASS
multi-cluster-gateway gke-l7-global-external-managed-mc
$ kubectl get gateway.networking.x-k8s.io
NAME CLASS
multi-cluster-gateway gke-l7-global-external-managed-mc
# Istio Gateway
$ kubectl get gw
NAME AGE
bookinfo-gateway 64m
$ kubectl get gateway.networking.istio.io
NAME AGE
bookinfo-gateway 64m
Troubleshooting
Proxy-only subnet missing in the region
Symptom:
The following issue might occur when you create a regional Gateway (internal or external):
generic::invalid_argument: error ensuring load balancer: Insert: Invalid value for field 'resource.target': 'regions/[REGION_NAME]/targetHttpProxies/gkegw-x5vt-default-internal-http-[ID]'. A reserved managed proxy subnetwork with purpose REGIONAL_MANAGED_PROXY is required.
Reason:
This error message indicates that no proxy-only subnet exists in the region for your Gateway.
Workaround:
To resolve this issue, configure a proxy-only subnet.
Proxy-only subnet already exists in the region with the wrong purpose
Symptom:
The following issue might occur when you create a proxy-only subnet for your regional Gateway (internal or external):
ERROR: (gcloud.compute.networks.subnets.create) Could not fetch resource:
- The resource 'projects/[PROJECT_NAME]/regions/[REGION_NAME]/subnetworks/[PROXY_ONLY_SUBNET_NAME]' already exists
Reason:
This error message indicates that you attempted to create a regional proxy-only subnet in a region that already has a proxy-only subnet.
Workaround:
To resolve this issue, use the following steps:
Check that a proxy-only subnet already exists in the region and verify that it has the correct purpose:
List your subnets to find which one is the proxy-only subnet in the region:
gcloud compute networks subnets list --regions=COMPUTE_REGION
Replace
COMPUTE_REGION
with the Compute Engine region where you want to create your regional Gateway.Describe your proxy-only subnet in the region to find its purpose:
gcloud compute networks subnets describe PROXY_ONLY_SUBNET \ --region COMPUTE_REGION | grep -E 'name|purpose'
Replace
PROXY_ONLY_SUBNET
with the proxy-only subnet.
GKE Gateway only supports
REGIONAL_MANAGED_PROXY
proxy-only subnets for regional Gateways (internal or regional).If the existing proxy-only subnet in the region was created with an
INTERNAL_HTTPS_LOAD_BALANCER
purpose, migrate its purpose toREGIONAL_MANAGED_PROXY
.
No healthy upstream
Symptom:
The following issue might occur when you create a Gateway but cannot access the backend services (503 response code):
no healthy upstream
Reason:
This error message indicates that the health check prober cannot find healthy backend services. It is possible that your backend services are healthy but you might need to customize the health checks.
Workaround:
To resolve this issue, customize your health check based on your application's
requirements (for example, /health
) using a HealthCheckPolicy
.
What's next
- Learn more about the Gateway controller.
- Learn how to Configure Gateway resources using Policies.