This page shows you how to configure the load balancer that Google Kubernetes Engine (GKE) creates when you deploy a Gateway in a GKE cluster.
When you deploy a Gateway, the GatewayClass configuration determines which load balancer GKE creates. This managed load balancer is pre-configured with default settings that you can modify using a Policy.
You can customize Gateway resources to fit with your infrastructure or application requirements by attaching Policies to Gateways, Services or ServiceImports. After you apply or modify a Policy, you don't need to delete or recreate your Gateway, Route, or Service resources, the Policy is processed by the Gateway controller and the underlying load balancer resource is (re)configured according to the (new) Policy.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
GKE Gateway controller requirements
- For Standard, GKE version 1.24 or later.
- For Autopilot, GKE version 1.26 or later.
- Google Cloud CLI version 407.0.0 or later.
- The Gateway API is supported on VPC-native clusters only.
- If you are using the internal GatewayClasses, you must enable a proxy-only subnet.
- Your cluster must have the
HttpLoadBalancing
add-on enabled. - If you are using Istio, you must upgrade Istio to one of the following
versions:
- 1.15.2 or later
- 1.14.5 or later
- 1.13.9 or later.
- If you are using Shared VPC, then in the host project, you need to assign the
Compute Network User
role to the GKE Service account for the service project.
Restrictions and Limitations
In addition to the GKE Gateway controller restrictions and limitations, the following limitations apply specifically to Policies applied on the Gateway resources:
GCPGatewayPolicy
resources can only be attached to agateway.networking.k8s.io
Gateway
.GCPGatewayPolicy
resources must exist in the same namespace as the targetGateway
.When using a single cluster Gateway,
GCPBackendPolicy
, andHealthCheckPolicy
resources must refer to aService
resource.
- When using a multi-cluster Gateway,
GCPBackendPolicy
, andHealthCheckPolicy
, resources must refer to aServiceImport
resource.
Only one
GCPGatewayPolicy
can be attached to a Service at any given time. When twoGCPGatewayPolicy
policies are created and target the sameService
orServiceImport
, the oldest policy will take precedence and the second one will fail to be attached.Hierarchical policies are not supported with GKE Gateway.
HealthCheckPolicy
, andGCPBackendPolicy
resources must exist in the same namespace as the targetService
orServiceImport
resource.GCPBackendPolicy
andHealthCheckPolicy
resources are structured in a way that they can reference only one backend service.GCPBackendPolicy
does not supportHEADER_FIELD
orHTTP_COOKIE
options for session affinity.
Configure global access for your regional internal Gateway
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
To enable global access with your internal Gateway, attach a policy to the Gateway resource.
The following GCPGatewayPolicy
manifest enables regional internal Gateway for
global access:
apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
name: my-gateway-policy
namespace: default
spec:
default:
allowGlobalAccess: true
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
Configure the region for your multi-cluster Gateway
This section describes a functionality that is available on GKE clusters running version 1.30.3-gke.1225000 or later.
If your fleet has clusters across multiple regions, you might need to deploy regional Gateways in different regions for a variety of use cases, for example, cross-region redundancy, low latency and data sovereignty. In your multi-cluster Gateway config cluster, you can specify the region in which you want to deploy the regional Gateways. If you don't specify a region, the default region is the config cluster's region.
To configure a region for your multi-cluster Gateway, use the region
field in the GCPGatewayPolicy
. In the following example, the Gateway is
configured in the us-central1
region:
apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
name: my-gateway-policy
namespace: default
spec:
default:
region: us-central1
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-regional-gateway
Configure SSL Policies to secure client-to-load-balancer traffic
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
To secure your client-to-load-balancer traffic, configure the SSL policy by adding
the name of your policy to the GCPGatewayPolicy
. By default, the Gateway does not
have any SSL Policy defined and attached.
Make sure that you create an SSL policy
prior to referencing the policy in your GCPGatewayPolicy
.
The following GCPGatewayPolicy
manifest specifies a security policy named
gke-gateway-ssl-policy
:
apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
name: my-gateway-policy
namespace: team1
spec:
default:
sslPolicy: gke-gateway-ssl-policy
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
Configure health checks
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
You can use a HealthCheckPolicy
to control the load balancer health check
settings. Each type of health check (http
, https
, grpc
, and http2
) has a
parameters that you can define. Google Cloud creates a unique health
check for each backend service for each GKE Service.
For your load balancer to function normally, you might need to configure a custom
HealthCheckPolicy
for your load balancer if your health check path isn't the
standard "/". This configuration is also necessary if the path requires special
headers or if you need to adjust the health check parameters. For example, if the
default request path is "/" but your service can't be accessed at that request
path and instead uses "/health" to report its health, then you must configure
requestPath
in your HealthCheckPolicy
accordingly.
The following HealthCheckPolicy
manifest shows all the fields available when
configuring a health check policy:
Service
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
name: lb-healthcheck
namespace: lb-service-namespace
spec:
default:
checkIntervalSec: INTERVAL
timeoutSec: TIMEOUT
healthyThreshold: HEALTHY_THRESHOLD
unhealthyThreshold: UNHEALTHY_THRESHOLD
logConfig:
enabled: ENABLED
config:
type: PROTOCOL
httpHealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
httpsHealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
grpcHealthCheck:
grpcServiceName: GRPC_SERVICE_NAME
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
http2HealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
name: lb-healthcheck
namespace: lb-service-namespace
spec:
default:
checkIntervalSec: INTERVAL
timeoutSec: TIMEOUT
healthyThreshold: HEALTHY_THRESHOLD
unhealthyThreshold: UNHEALTHY_THRESHOLD
logConfig:
enabled: ENABLED
config:
type: PROTOCOL
httpHealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
httpsHealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
grpcHealthCheck:
grpcServiceName: GRPC_SERVICE_NAME
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
http2HealthCheck:
portSpecification: PORT_SPECIFICATION
port: PORT
portName: PORT_NAME
host: HOST
requestPath: REQUEST_PATH
response: RESPONSE
proxyHeader: PROXY_HEADER
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
Replace the following:
INTERVAL
: specifies the check-interval, in seconds, for each health check prober. This is the time from the start of one prober's check to the start of its next check. If you omit this parameter, the Google Cloud default is 15 seconds if noHealthCheckPolicy
is specified, and is 5 seconds when aHealthCheckPolicy
is specified with nocheckIntervalSec
value. For more information, see Multiple probes and frequency.TIMEOUT
: specifies the amount of time that Google Cloud waits for a response to a probe. The value ofTIMEOUT
must be less than or equal to theINTERVAL
. Units are seconds. Each probe requires an HTTP 200 (OK) response code to be delivered before the probe timeout.HEALTHY_THRESHOLD
andUNHEALTHY_THRESHOLD
: specifies the number of sequential connection attempts that must succeed or fail, for at least one prober, to change the health state from healthy to unhealthy or unhealthy to healthy. If you omit one of these parameters, the Google Cloud default is 2.PROTOCOL
: specifies a protocol used by probe systems for health checking. For more information, see Success criteria for HTTP, HTTPS, and HTTP/2 and Success criteria for gRPC. This parameter is required.ENABLED
: specifies if logging is enabled or disabled.PORT_SPECIFICATION
: specifies if the health check uses a fixed port (USE_FIXED_PORT
), named port (USE_NAMED_PORT
) or serving port (USE_SERVING_PORT
). If not specified, the health check follows the behavior specified in theport
andportName
fields. If neitherport
orportName
are specified, this field defaults toUSE_SERVING_PORT
.PORT
: A HealthCheckPolicy only supports specifying the load balancer health check port by using a port number. If you omit this parameter, the Google Cloud default is 80. Because the load balancer sends probes to the Pod's IP address directly, you should select a port matching acontainerPort
of a serving Pods, even if thecontainerPort
is referenced by atargetPort
of the Service. You are not limited tocontainerPorts
referenced by a Service'stargetPort
.PORT_NAME
: specifies the port name as defined in InstanceGroup.NamedPort.name. If bothport
andportName
are defined, Google Cloud considers theport
value first.HOST
: the value of the host header in the health check request. This value uses the RFC 1123 definition of a hostname except numeric IP addresses are not allowed. If not specified or left empty, this value defaults to the IP address of the health check.REQUEST_PATH
: specifies the request-path of the health check request. If not specified or left empty, defaults to/
.RESPONSE
: specifies the bytes to match against the beginning of the response data. If not specified or left empty, GKE interprets any response as healthy. The response data can only be ASCII.PROXY_HEADER
: specifies the proxy header type. You can useNONE
orPROXY_V1
. Defaults toNONE
.GRPC_SERVICE_NAME
: an optional name of the gRPC Service. Omit this field to specify all Services.
For more information about HealthCheckPolicy fields, see the healthChecks
reference.
Configure Google Cloud Armor backend security policy to secure your backend services
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
Configure the Google Cloud Armor backend security policy by adding the name of
your security policy to the GCPBackendPolicy
to secure your backend services.
By default, the Gateway does not have any Google Cloud Armor backend security
policy defined and attached.
Make sure that you create a Google Cloud Armor backend security
policy prior to referencing the policy
in your GCPBackendPolicy
. If you are enabling a regional Gateway, then you
must create a regional Google Cloud Armor backend security policy.
The following GCPBackendPolicy
manifest specifies a backend security policy
named example-security-policy
:
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
securityPolicy: example-security-policy
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
securityPolicy: example-security-policy
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
Configure IAP
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
Identity-Aware Proxy (IAP) enforces access control policies on backend services associated with an HTTPRoute. With this enforcement, only authenticated users or applications with the correct Identity and Access Management (IAM) role assigned can access these backend services.
By default, there is no IAP applied to your backend services, you
need to explicitly configure IAP in a GCPBackendPolicy
.
To configure IAP with Gateway, do the following:
Enable IAP for GKE Do not configure the backend (Configuring BackendConfig) because
BackendConfig
is only valid in the case of an Ingress deployment.Create a secret for your IAP:
In the Google Cloud console, go to the Credentials page:
Click the name of the client and download the OAuth client file.
From the OAuth client file, copy the OAuth secret on the clipboard.
Create a file called
iap-secret.txt
.Paste the OAuth secret in the
iap-secret.txt
file using the following command:echo -n CLIENT_SECRET > iap-secret.txt kubectl create secret generic SECRET_NAME --from-file=key=iap-secret.txt
To specify IAP policy referencing a secret:
Create the following
GCPBackendPolicy
manifest, replace theSECRET_NAME
andCLIENT_ID
respectively. Save the manifest asbackend-policy.yaml
:Service
apiVersion: networking.gke.io/v1 kind: GCPBackendPolicy metadata: name: backend-policy spec: default: iap: enabled: true oauth2ClientSecret: name: SECRET_NAME clientID: CLIENT_ID targetRef: group: "" kind: Service name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1 kind: GCPBackendPolicy metadata: name: backend-policy spec: default: iap: enabled: true oauth2ClientSecret: name: SECRET_NAME clientID: CLIENT_ID targetRef: group: net.gke.io kind: ServiceImport name: lb-service
Apply the
backend-policy.yaml
manifest:kubectl apply -f backend-policy.yaml
Verify your configuration:
Confirm that the policy was applied after creating your
GCPBackendPolicy
with IAP:kubectl get gcpbackendpolicy
The output is similar to the following:
NAME AGE backend-policy 45m
To get more details, use the describe command:
kubectl describe gcpbackendpolicy
The output is similar to the following:
Name: backend-policy Namespace: default Labels: <none> Annotations: <none> API Version: networking.gke.io/v1 Kind: GCPBackendPolicy Metadata: Creation Timestamp: 2023-05-27T06:45:32Z Generation: 2 Resource Version: 19780077 UID: f4f60a3b-4bb2-4e12-8748-d3b310d9c8e5 Spec: Default: Iap: Client ID: 441323991697-luotsrnpboij65ebfr13hlcpm5a4heke.apps.googleusercontent.com Enabled: true oauth2ClientSecret: Name: my-iap-secret Target Ref: Group: Kind: Service Name: lb-service Status: Conditions: Last Transition Time: 2023-05-27T06:48:25Z Message: Reason: Attached Status: True Type: Attached Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 46m sc-gateway-controller default/backend-policy Normal SYNC 44s (x15 over 43m) sc-gateway-controller Application of GCPGatewayPolicy "default/backend-policy" was a success
Configure backend service timeout
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
The following GCPBackendPolicy
manifest specifies a
backend service timeout
period of 40 seconds. The timeoutSec
field defaults to 30 seconds.
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
timeoutSec: 40
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
timeoutSec: 40
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
Configure session affinity
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
You can configure session affinity based on the following criteria:
- Client IP address
- Generated cookie
When you configure session affinity
for your Service, the Gateway's localityLbPolicy
setting is set to MAGLEV
.
When you remove a session affinity configuration from the GCPBackendPolicy
,
the Gateway reverts the localityLbPolicy
setting to the default value, ROUND_ROBIN
.
The following GCPBackendPolicy
manifest specifies a session affinity based on the
client IP address:
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
sessionAffinity:
type: CLIENT_IP
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
sessionAffinity:
type: CLIENT_IP
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
The following GCPBackendPolicy
manifest specifies a session affinity based on a
generated cookie
and configures the cookies TTL to 50 seconds:
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
sessionAffinity:
type: GENERATED_COOKIE
cookieTtlSec: 50
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
sessionAffinity:
type: GENERATED_COOKIE
cookieTtlSec: 50
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
You can use the following values for the sessionAffinity.type
field:
CLIENT_IP
GENERATED_COOKIE
NONE
Configure connection draining timeout
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
You can configure
connection draining timeout
using GCPBackendPolicy
. Connection draining timeout is the time, in seconds, to
wait for connections to drain. The timeout duration can be from 0 to 3600 seconds.
The default value is 0, which also disables connection draining.
The following GCPBackendPolicy
manifest specifies a connection draining timeout
of 60 seconds:
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
connectionDraining:
drainingTimeoutSec: 60
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
connectionDraining:
drainingTimeoutSec: 60
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
For the specified duration of the timeout, GKE waits for existing requests to the removed backend to complete. The load balancer does not send new requests to the removed backend. After the timeout duration is reached, GKE closes all remaining connections to the backend.
HTTP access logging
This section describes a functionality that is available on GKE clusters running version 1.24 or later.
By default:
- The Gateway controller logs all HTTP requests from clients to Cloud Logging.
- The sampling rate is 1,000,000, which means all requests are logged.
You can disable access logging on your Gateway using a GCPBackendPolicy
in three ways:
- You can leave the
GCPBackendPolicy
with nologging
section - You can set
logging.enabled
tofalse
- You can set
logging.enabled
totrue
and setlogging.sampleRate
to0
You can also configure the access logging sampling rate.
The following GCPBackendPolicy
manifest modifies access logging's default sample rate and sets it to 50% of the HTTP requests for a given Service resource:
Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
logging:
enabled: true
sampleRate: 500000
targetRef:
group: ""
kind: Service
name: lb-service
Multi-cluster Service
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: my-backend-policy
namespace: lb-service-namespace
spec:
default:
logging:
enabled: true
sampleRate: 500000
targetRef:
group: net.gke.io
kind: ServiceImport
name: lb-service
This manifest has the following fields:
enable: true
: explicitly enables access logging. Logs are available in Logging.sampleRate: 500000
: specifies that 50% of packets are logged. You can use a value between 0 and 1,000,000. GKE converts this value to a floating point value in the range [0, 1] by dividing by 1,000,000. This field is only relevant ifenable
is set totrue
.sampleRate
is an optional field, but if it's configured thenenable: true
must also be set. Ifenable
is set totrue
andsampleRate
is not provided then GKE setsenable
tofalse
.
Configure traffic-based autoscaling for your single-cluster Gateway
Ensure your GKE cluster is running the version 1.31.1-gke.2008000 or later.
To enable traffic-based autoscaling and capacity-based load balancing in a single-cluster Gateway, you can configure Service capacity. Service capacity is the ability to specify the amount of traffic capacity that a Service can receive before Pods are autoscaled or traffic overflows to other available clusters.
To configure Service capacity, create a Service and an associated
GCPBackendPolicy
. The GCPBackendPolicy
manifest uses the field
maxRatePerEndpoint
which defines a maximum Requests per Second (RPS) value per
Pod in a Service. The following GCPBackendPolicy
manifest defines a maximum
RPS of 10:
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
name: store
spec:
default:
maxRatePerEndpoint: 10
targetRef:
group: ""
kind: Service
name: store
To learn more about traffic-based autoscaling, see Autoscaling based on load balancer traffic.
Troubleshooting
Multiple GCPBackendPolicy
attached to the same Service
Symptom:
The following status condition might occur when you attach a GCPBackendPolicy
to a Service
or a ServiceImport
:
status:
conditions:
- lastTransitionTime: "2023-09-26T20:18:03Z"
message: conflicted with GCPBackendPolicy "[POLICY_NAME]" of higher precedence, hence not applied
reason: Conflicted
status: "False"
type: Attached
Reason:
This status condition indicates that you are trying to apply a second GCPBackendPolicy
to a Service
or ServiceImport
that already has a GCPBackendPolicy
attached.
Multiple GCPBackendPolicy
attached to the same Service
or ServiceImport
is
not supported with GKE Gateway. See Restrictions and Limitations
for more details.
Workaround:
Configure a single GCPBackendPolicy
that includes all custom configurations and
attach it to your Service
or ServiceImport
.
Google Cloud Armor security policy not found
Symptom:
The following error message might appear when you enable Google Cloud Armor on your regional Gateway:
Invalid value for field 'resource': '{
"securityPolicy":"projects/123456789/regions/us-central1/securityPolicies/<policy_name>"}'.
The given security policy does not exist.
Reason:
The error message indicates that the specified regional Google Cloud Armor security policy does not exist in your Google Cloud project.
Workaround:
Create a regional Google Cloud Armor security
policy in your project and reference
this policy in your GCPBackendPolicy
.
What's next
- Learn more about the Gateway controller.
- Learn how to reference a Gateway from a resource.
- View the Policy Types API reference.
- View the API type definitions.