Configure Gateway resources using Policies

Autopilot Standard

---

This page shows you how to configure the load balancer that Google Kubernetes Engine (GKE) creates when you deploy a Gateway in a GKE cluster.

When you deploy a Gateway, the GatewayClass configuration determines which load balancer GKE creates. This managed load balancer is pre-configured with default settings that you can modify using a Policy.

You can customize Gateway resources to fit with your infrastructure or application requirements by attaching Policies to Gateways, Services or ServiceImports. After you apply or modify a Policy, you don't need to delete or recreate your Gateway, Route, or Service resources, the Policy is processed by the Gateway controller and the underlying load balancer resource is (re)configured according to the (new) Policy.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

GKE Gateway controller requirements

• For Standard, GKE version 1.24 or later.

• For Autopilot, GKE version 1.26 or later.
• Google Cloud CLI version 407.0.0 or later.
• The Gateway API is supported on VPC-native clusters only.
• If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
• Your cluster must have the HttpLoadBalancing add-on enabled.
• If you are using Istio, you must upgrade Istio to one of the following versions:
  • 1.15.2 or later
  • 1.14.5 or later
  • 1.13.9 or later.
• If you are using Shared VPC, then in the host project, you need to assign the Compute Network User role to the GKE Service account for the service project.

Restrictions and Limitations

In addition to the GKE Gateway controller restrictions and limitations, the following limitations apply specifically to Policies applied on the Gateway resources:

• GCPGatewayPolicy resources can only be attached to a gateway.networking.k8s.io Gateway.

• GCPGatewayPolicy resources must exist in the same namespace as the target Gateway.

• When using a single cluster Gateway, GCPBackendPolicy, and HealthCheckPolicy resources must refer to a Service resource.

• When using a multi-cluster Gateway, GCPBackendPolicy, and HealthCheckPolicy, resources must refer to a ServiceImport resource.

• Only one GCPGatewayPolicy can be attached to a Service at any given time. When two GCPGatewayPolicy policies are created and target the same Service or ServiceImport, the oldest policy will take precedence and the second one will fail to be attached.

• Hierarchical policies are not supported with GKE Gateway.

• HealthCheckPolicy, and GCPBackendPolicy resources must exist in the same namespace as the target Service or ServiceImport resource.

• GCPBackendPolicy and HealthCheckPolicy resources are structured in a way that they can reference only one backend service.

Configure global access for your regional internal Gateway

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

To enable global access with your internal Gateway, attach a policy to the Gateway resource.

The following GCPGatewayPolicy manifest enables regional internal Gateway for global access:

apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
  name: my-gateway-policy
  namespace: default
spec:
  default:
    allowGlobalAccess: true
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway

Note: Upgrading an existing internal Gateway by adding global access recreates the forwarding rule of your regional internal load balancer. This upgrade deletes and then re-creates Google Cloud load balancers which results in up to 15 minutes of unavailability. We recommend performing this operation during a maintenance window to avoid any application downtime.

Configure SSL Policies to secure client-to-load-balancer traffic

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

To secure your client-to-load-balancer traffic, configure the SSL policy by adding the name of your policy to the GCPGatewayPolicy. By default, the Gateway does not have any SSL Policy defined and attached.

Make sure that you create an SSL policy prior to referencing the policy in your GCPGatewayPolicy.

Note: If you use a regional Gateway, make sure you create a regional SSL policy.

The following GCPGatewayPolicy manifest specifies a security policy named gke-gateway-ssl-policy:

apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
  name: my-gateway-policy
  namespace: team1
spec:
  default:
    sslPolicy: gke-gateway-ssl-policy
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway

Configure health checks

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

You can use a HealthCheckPolicy to control the load balancer health check settings. Each type of health check (http, https, grpc, and http2) has a parameters that you can define. Google Cloud creates a unique health check for each backend service for each GKE Service.

The following HealthCheckPolicy manifest shows all the fields available when configuring a health check policy:

Service

apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: lb-healthcheck
  namespace: lb-service-namespace
spec:
  default:
    checkIntervalSec: INTERVAL
    timeoutSec: TIMEOUT
    healthyThreshold: HEALTHY_THRESHOLD
    unhealthyThreshold: UNHEALTHY_THRESHOLD
    logConfig:
      enabled: ENABLED
    config:
      type: PROTOCOL
      httpHealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
      httpsHealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
      grpcHealthCheck:
        grpcServiceName: GRPC_SERVICE_NAME
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
      http2HealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: lb-healthcheck
  namespace: lb-service-namespace
spec:
  default:
    checkIntervalSec: INTERVAL
    timeoutSec: TIMEOUT
    healthyThreshold: HEALTHY_THRESHOLD
    unhealthyThreshold: UNHEALTHY_THRESHOLD
    logConfig:
      enabled: ENABLED
    config:
      type: PROTOCOL
      httpHealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
      httpsHealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
      grpcHealthCheck:
        grpcServiceName: GRPC_SERVICE_NAME
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
      http2HealthCheck:
        portSpecification: PORT_SPECIFICATION
        port: PORT
        portName: PORT_NAME
        host: HOST
        requestPath: REQUEST_PATH
        response: RESPONSE
        proxyHeader: PROXY_HEADER
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

Replace the following:

• INTERVAL: specifies the check-interval, in seconds, for each health check prober. This is the time from the start of one prober's check to the start of its next check. If you omit this parameter, the Google Cloud default is 5 seconds. For more information, see Multiple probes and frequency.
• TIMEOUT: specifies the amount of time that Google Cloud waits for a response to a probe. The value of TIMEOUT must be less than or equal to the INTERVAL. Units are seconds. Each probe requires an HTTP 200 (OK) response code to be delivered before the probe timeout.
• HEALTHY_THRESHOLD and UNHEALTHY_THRESHOLD: specifies the number of sequential connection attempts that must succeed or fail, for at least one prober, to change the health state from healthy to unhealthy or unhealthy to healthy. If you omit one of these parameters, the Google Cloud default is 2.
• PROTOCOL: specifies a protocol used by probe systems for health checking. For more information, see Success criteria for HTTP, HTTPS, and HTTP/2 and Success criteria for gRPC. This parameter is required.
• ENABLED: specifies if logging is enabled or disabled.
• PORT_SPECIFICATION: specifies if the health check uses a fixed port (USE_FIXED_PORT), named port (USE_NAMED_PORT) or serving port (USE_SERVING_PORT). If not specified, the health check follows the behavior specified in the port and portName fields. If neither port or portName are specified, this field defaults to USE_SERVING_PORT.
• PATH: specifies the request-path to which the probe system should connect for HTTP, HTTPS, or HTTP2 health checks. If you omit this parameter, the Google Cloud default is /.
• PORT: A HealthCheckPolicy only supports specifying the load balancer health check port by using a port number. If you omit this parameter, the Google Cloud default is 80. Because the load balancer sends probes to the Pod's IP address directly, you should select a port matching a containerPort of a serving Pods, even if the containerPort is referenced by a targetPort of the Service. You are not limited to containerPorts referenced by a Service's targetPort.
• PORT_NAME: specifies the port name as defined in InstanceGroup.NamedPort.name. If both port and portName are defined, Google Cloud considers the port value first.
• HOST: the value of the host header in the health check request. This value uses the RFC 1123 definition of a hostname except numeric IP addresses are not allowed. If not specified or left empty, this value defaults to the IP address of the health check.
• REQUEST_PATH: specifies the request path of the health check request. If not specified or left empty, defaults to /.
• RESPONSE: specifies the bytes to match against the beginning of the response data. If not specified or left empty, GKE interprets any response as healthy. The response data can only be ASCII.
• PROXY_HEADER: specifies the proxy header type. You can use NONE or PROXY_V1. Defaults to NONE.
• GRPC_SERVICE_NAME: an optional name of the gRPC Service. Omit this field to specify all Services.

For more information about HealthCheckPolicy fields, see the healthChecks reference.

Configure Google Cloud Armor backend security policy to secure your backend services

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Note: Google Cloud has replaced the LbPolicy policy with the GCPBackendPolicy policy. The LbPolicy is still supported, but with no additional attributes.

Configure the Google Cloud Armor backend security policy by adding the name of your security policy to the GCPBackendPolicy to secure your backend services. By default, the Gateway does not have any Google Cloud Armor backend security policy defined and attached.

Make sure that you create a Google Cloud Armor backend security policy prior to referencing the policy in your GCPBackendPolicy.

The following GCPBackendPolicy manifest specifies a backend security policy named example-security-policy:

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    securityPolicy: example-security-policy
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    securityPolicy: example-security-policy
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

Configure IAP

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Identity-Aware Proxy (IAP) enforces access control policies on backend services associated with an HTTPRoute. With this enforcement, only authenticated users or applications with the correct Identity and Access Management (IAM) role assigned can access these backend services.

By default, there is no IAP applied to your backend services, you need to explicitly configure IAP in a GCPBackendPolicy.

To configure IAP with Gateway, do the following:

1. Enable IAP for GKE Do not configure the backend (Configuring BackendConfig) because BackendConfig is only valid in the case of an Ingress deployment.

2. Create a secret for your IAP:

   1. Go to the Credentials page. Button: Go to the Credentials page.

   2. Click the name of the client and download the OAuth client file.

   3. From the OAuth client file, copy the OAuth secret on the clipboard.

   4. Create a file called iap-secret.txt.

   5. Paste the OAuth secret in the iap-secret.txt file using the following command:

      echo -n CLIENT_SECRET > iap-secret.txt
      kubectl create secret generic SECRET_NAME --from-file=key=iap-secret.txt
      

3. To specify IAP policy referencing a secret:

   1. Create the following GCPBackendPolicy manifest, replace the SECRET_NAME and CLIENT_ID respectively. Save the manifest as backend-policy.yaml:

      Service

      apiVersion: networking.gke.io/v1
      kind: GCPBackendPolicy
      metadata:
        name: backend-policy
      spec:
        default:
          iap:
            enabled: true
            oauth2ClientSecret:
              name: SECRET_NAME
            clientID: CLIENT_ID
        targetRef:
          group: ""
          kind: Service
          name: lb-service
      

      Multi-cluster Service

      apiVersion: networking.gke.io/v1
      kind: GCPBackendPolicy
      metadata:
        name: backend-policy
      spec:
        default:
          iap:
            enabled: true
            oauth2ClientSecret:
              name: SECRET_NAME
            clientID: CLIENT_ID
        targetRef:
          group: net.gke.io
          kind: ServiceImport
          name: lb-service
      

   2. Apply the backend-policy.yaml manifest:

      kubectl apply -f backend-policy.yaml
      

4. Verify your configuration:

   1. Confirm that the policy was applied after creating your GCPBackendPolicy with IAP:

      kubectl get gcpbackendpolicy
      

      The output is similar to the following:

      NAME             AGE
      backend-policy   45m
      

   2. To get more details, use the describe command:

      kubectl describe gcpbackendpolicy
      

      The output is similar to the following:

      Name:         backend-policy
      Namespace:    default
      Labels:       <none>
      Annotations:  <none>
      API Version:  networking.gke.io/v1
      Kind:         GCPBackendPolicy
      Metadata:
        Creation Timestamp:  2023-05-27T06:45:32Z
        Generation:          2
        Resource Version:    19780077
        UID:                 f4f60a3b-4bb2-4e12-8748-d3b310d9c8e5
      Spec:
        Default:
          Iap:
            Client ID:  441323991697-luotsrnpboij65ebfr13hlcpm5a4heke.apps.googleusercontent.com
            Enabled:    true
            oauth2ClientSecret:
              Name:  my-iap-secret
        Target Ref:
          Group:
          Kind:   Service
          Name:   lb-service
      Status:
        Conditions:
          Last Transition Time:  2023-05-27T06:48:25Z
          Message:
          Reason:                Attached
          Status:                True
          Type:                  Attached
      Events:
        Type     Reason  Age                 From                   Message
        ----     ------  ----                ----                   -------
        Normal   ADD     46m                 sc-gateway-controller  default/backend-policy
        Normal   SYNC    44s (x15 over 43m)  sc-gateway-controller  Application of GCPGatewayPolicy "default/backend-policy" was a success
      

Configure backend service timeout

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Note: Google Cloud has replaced the LbPolicy policy with the GCPBackendPolicy policy. The LbPolicy is still supported, but with no additional attributes.

The following GCPBackendPolicy manifest specifies a backend service timeout period of 40 seconds. The timeoutSec field defaults to 30 seconds.

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    timeoutSec: 40
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    timeoutSec: 40
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

Configure session affinity

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Note: Google Cloud has replaced the LbPolicy policy with the GCPBackendPolicy policy. The LbPolicy is still supported, but with no additional attributes.

You can configure session affinity based on the following criteria:

• Client IP address
• Generated cookie

When you configure session affinity for your Service, the Gateway's localityLbPolicy setting is set to MAGLEV.

When you remove a session affinity configuration from the GCPBackendPolicy, the Gateway reverts the localityLbPolicy setting to the default value, ROUND_ROBIN. This value is silently set on the GKE-managed backend service (attached to the load balancer) and is not reflected in the output of a gcloud CLI command, in the UI, or with Terraform.

The following GCPBackendPolicy manifest specifies a session affinity based on the client IP address:

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    sessionAffinity:
      type: CLIENT_IP
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    sessionAffinity:
      type: CLIENT_IP
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

The following GCPBackendPolicy manifest specifies a session affinity based on a generated cookie and configures the cookies TTL to 50 seconds:

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    sessionAffinity:
      type: GENERATED_COOKIE
      cookieTtlSec: 50
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    sessionAffinity:
      type: GENERATED_COOKIE
      cookieTtlSec: 50
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

You can use the following options for sessionAffinity.type: CLIENT_IP, CLIENT_IP_PROTO, CLIENT_IP_PORT_PROTO, GENERATED_COOKIE, HEADER_FIELD, HTTP_COOKIE, NONE.

Configure connection draining timeout

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Note: Google Cloud has replaced the LbPolicy policy with the GCPBackendPolicy policy. The LbPolicy is still supported, but with no additional attributes.

You can configure connection draining timeout using GCPBackendPolicy. Connection draining timeout is the time, in seconds, to wait for connections to drain. The timeout duration can be from 0 to 3600 seconds. The default value is 0, which also disables connection draining.

The following GCPBackendPolicy manifest specifies a connection draining timeout of 60 seconds:

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    connectionDraining:
      drainingTimeoutSec: 60
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    connectionDraining:
      drainingTimeoutSec: 60
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

For the specified duration of the timeout, GKE waits for existing requests to the removed backend to complete. The load balancer does not send new requests to the removed backend. After the timeout duration is reached, GKE closes all remaining connections to the backend.

HTTP access logging

This section describes a functionality that is available on GKE clusters running version 1.24 or later.

Note: Google Cloud has replaced the LbPolicy policy with the GCPBackendPolicy policy. The LbPolicy is still supported, but with no additional attributes.

By default:

• The Gateway controller logs all HTTP requests from clients to Cloud Logging.
• The sampling rate is 1,000,000, which means all requests are logged.

You can disable access logging on your Gateway using a GCPBackendPolicy in three ways:

• You can leave the GCPBackendPolicy with no logging section
• You can set logging.enabled to false
• You can set logging.enabled to true and set logging.sampleRate to 0

You can also configure the access logging sampling rate.

The following GCPBackendPolicy manifest modifies access logging's default sample rate and sets it to 50% of the HTTP requests for a given Service resource:

Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    logging:
      enabled: true
      sampleRate: 500000
  targetRef:
    group: ""
    kind: Service
    name: lb-service

Multi-cluster Service

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: my-backend-policy
  namespace: lb-service-namespace
spec:
  default:
    logging:
      enabled: true
      sampleRate: 500000
  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: lb-service

This manifest has the following fields:

• enable: true: explicitly enables access logging. Logs are available in Logging.
• sampleRate: 500000: specifies that 50% of packets are logged. You can use a value between 0 and 1,000,000. GKE converts this value to a floating point value in the range [0, 1] by dividing by 1,000,000. This field is only relevant if enable is set to true. sampleRate is an optional field, but if it's configured then enable: true must also be set. If enable is set to true and sampleRate is not provided then GKE sets enable to false.

Troubleshooting

Multiple GCPBackendPolicy attached to the same Service

Symptom:

The following status condition might occur when you attach a GCPBackendPolicy to a Service or a ServiceImport:

status:
  conditions:
    - lastTransitionTime: "2023-09-26T20:18:03Z"
      message: conflicted with GCPBackendPolicy "[POLICY_NAME]" of higher precedence, hence not applied
      reason: Conflicted
      status: "False"
      type: Attached

Reason:

This status condition indicates that you are trying to apply a second GCPBackendPolicy to a Service or ServiceImport that already has a GCPBackendPolicy attached.

Multiple GCPBackendPolicy attached to the same Service or ServiceImport is not supported with GKE Gateway. See Restrictions and Limitations for more details.

Workaround:

Configure a single GCPBackendPolicy that includes all custom configurations and attach it to your Service or ServiceImport.

What's next

• Learn more about the Gateway controller.
• Learn how to reference a Gateway from a resource.