Configuring Ingress features

This page provides a comprehensive overview of what's supported and configurable through Kubernetes Ingress on Google Cloud. Ingress for Google Kubernetes Engine (GKE) and Anthos provides enterprise-class load balancing with tight integration to your Google Cloud VPC network.

Feature comparison

The following table provides a list of supported features for Ingress on Google Cloud. The availability of the feature, General availability (GA) or Beta is also indicated.

Ingress class	External Ingress	Internal Ingress	Multi-cluster Ingress
Ingress controller	Cluster-hosted Ingress controller		Google-hosted Ingress controller
Google Cloud load balancer type	External HTTP(S) LB	Internal HTTP(S) LB	External HTTP(S) LB
Cluster scope	Single cluster	Single cluster	Multi-cluster
Load balancer scope	Global	Regional	Global
Version support	All	GKE 1.16.5+	GKE 1.14+
Environment support	GKE	GKE	GKE
Shared VPC support	GA	GA	GA
Service annotations
Container-native Load Balancing (NEGs)	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
HTTPS from load balancer to backends	GA	GA	GA
HTTP/2	GA	GA TLS Only	GA
Ingress annotations
Static IP addresses	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Kubernetes Secrets-based certificates	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Google self-managed SSL certificates	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Google-managed SSL certificates	GA 1.17.9-gke.6300^G All^B
FrontendConfig (beta)
SSL policy	GA 1.16.15-gke.4901^G
HTTP-to-HTTPS redirect	Beta 1.18.10-gke.600^B
BackendConfig
Backend service timeout	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Cloud CDN	GA		GA
Connection draining timeout	GA	GA	GA
Custom load balancer health check configuration	GA 1.16.15-gke.4901^G	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Google Cloud Armor security policy	Beta		Beta
HTTP access logging configuration	GA 1.16.10-gke.6^G	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
Identity-Aware Proxy (IAP)	GA		GA
Session affinity	GA	GA 1.16.5.x^B 1.17.13-gke.2600^G	GA
User-defined request headers	GA 1.15.3-gke.1^G		GA

^BThis feature is available in beta starting from the specified version. Features without a version listed are supported for all available GKE and Anthos versions.

^GThis feature is supported as GA starting from the specified version.

Configuring Ingress features

When creating an Ingress using the default controller, you can choose the type of load balancer (an external HTTP(S) load balancer or an internal HTTP(S) load balancer) by using an annotation on the Ingress object. You can choose whether GKE creates zonal NEGs or if it uses instance groups by using an annotation on each Service object.

FrontendConfig and BackendConfig custom resource definitions (CRDs) allow you to further customize the load balancer. These CRDs allow you to define additional load balancer features hierarchically, in a more structured way than annotations. To use Ingress (and these CRDs), you must have the HTTP load balancing add-on enabled. GKE clusters have HTTP load balancing enabled by default; you must not disable it.

FrontendConfigs are referenced in an Ingress object and BackendConfigs are referenced by a Service object. The same CRDs can be referenced by multiple Service or Ingress objects for configuration consistency. The FrontendConfig and BackendConfig CRDs share the same lifecycle as their corresponding Ingress and Service resources and they are often deployed together.

The following diagram illustrates how:

• An annotation on an Ingress or MultiClusterIngress object references a FrontendConfig CRD. The FrontendConfig CRD references a Google Cloud SSL Policy.

• An annotation on a Service or MultiClusterService object references a BackendConfig CRD. The BackendConfig CRD specifies custom settings for the corresponding backend service's health check.

Figure: BackendConfig and FrontendConfig overview

Associating FrontendConfig with your Ingress

A FrontendConfig CRD is referenced by an annotation on an Ingress or MultiClusterIngress resource. The FrontendConfig corresponds to the entire Ingress/MultiClusterIngress resource, as shown in the following example:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    networking.gke.io/v1beta1.FrontendConfig: "frontendconfig"
...

Associating BackendConfig with your Ingress

A BackendConfig CRD is referenced by an annotation on a Service or MultiClusterService resource. The Service or MultiClusterService uses the cloud.google.com/backend-config annotation to specify the name of the BackendConfig. When associated with a Service or MultiClusterService the BackendConfig can determine the Google Cloud backend service settings. The following example demonstrates how to attach a BackendConfig to a Service or MultiClusterService. The default key means that all ports within the Service are associated with the my-backendconfig BackendConfig when those Service ports are referenced by an Ingress:

Important: If you are using GKE version 1.16-gke.3 or later we recommend you use the cloud.google.com/backend-config annotation. If you are using an earlier version, use the beta.cloud.google.com/backend-config annotation instead.

1.16-gke.3+

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "my-backendconfig"}'
...

All supported versions

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "my-backendconfig"}'
...

If you require unique BackendConfig settings for different ports within the same Service it's possible to associate specific BackendConfig resources to those ports explicitly. The ports key in the following example allows for the explicit linking of ports:

1.16-gke.3+

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"ports": {
    "port-1":"backendconfig-name-1",
    "port-2":"backendconfig-name-2"
    }}'
spec:
  ports:
  - name:service-name-1
    port: service-port-1
    protocol: TCP
    targetPort: 50000
  - name: service-name-2
    port: service-port-2
    protocol: TCP
    targetPort: 8080
...

All supported versions

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {
    "port-1":"backendconfig-name-1",
    "port-2":"backendconfig-name-2"
    }}'
spec:
  ports:
  - name: service-name-1
    port:  service-port-1
    protocol: TCP
    targetPort: 50000
  - name: service-name-2
    port: >service-port-2
    protocol: TCP
    targetPort:8080
...

port-1 can refer to ports within the Service by number, through service-port-1, or by name, through service-name-1 (only supported on GKE). Each port maps to a Google Cloud load balancer backend service which allows ports within the Service to have different configurations.

Configuring Ingress features through FrontendConfig parameters

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

The following section shows you how to set your FrontendConfig to enable specific Ingress features.

SSL policies

SSL policies allow you to specify a set of TLS versions and ciphers that the load balancer uses to terminate HTTPS traffic from clients. You must first create an SSL policy outside of GKE. Once created, you can reference it in a FrontendConfig CRD.

The sslPolicy field in the FrontendConfig references the name of an SSL policy in the same Google Cloud project as the GKE cluster. It attaches the SSL policy to the target HTTPS proxy, which was created for the external HTTP(S) load balancer by the Ingress. The same FrontendConfig resource and SSL policy can be referenced by multiple Ingress resources. If a referenced SSL policy is changed, the change is propagated to the Google Front Ends (GFEs) that power your external HTTP(S) load balancer created by the Ingress.

The following FrontendConfig manifest enables an SSL policy named gke-ingress-ssl-policy:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: my-frontend-config
spec:
  sslPolicy: gke-ingress-ssl-policy

HTTP to HTTPS redirects

An external HTTP load balancer can redirect unencrypted HTTP requests to an HTTPS load balancer that uses the same IP address. When you create an Ingress with HTTP to HTTPS redirects enabled, both of these load balancers are created automatically. Requests to the external IP address of the Ingress on port 80 are automatically redirected to the same external IP address on port 443. This functionality is built on HTTP to HTTPS redirects provided by Cloud Load Balancing.

To support HTTP to HTTPS redirection, an Ingress must be configured to serve both HTTP and HTTPS traffic. If either HTTP or HTTPS is disabled, redirection will not work.

HTTP to HTTPS redirects are configured using the redirectToHttps field in a FrontendConfig CRD. Redirects are enabled for the entire Ingress resource so all Services referenced by the Ingress will have HTTPS redirects enabled.

The following FrontendConfig manifest enables HTTP to HTTPS redirects. Set the spec.redirectToHttps.enabled field to true to enable HTTPS redirects. The spec.responseCodeName field is optional. If it's omitted a 301 Moved Permanently redirect is used.

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: my-frontend-config
spec:
  redirectToHttps:
    enabled: true
    responseCodeName: response-code

Replace response-code with one of the following:

• MOVED_PERMANENTLY_DEFAULT to return a 301 redirect response code (default if responseCodeName is unspecified).
• FOUND to return a 302 redirect response code.
• SEE_OTHER to return a 303 redirect response code.
• TEMPORARY_REDIRECT to return a 307 redirect response code.
• PERMANENT_REDIRECT to return a 308 redirect response code.

When redirects are enabled the Ingress controller creates a load balancer as described in Setting up HTTP-to-HTTPS redirect for external HTTP(S) load balancers and as shown in this diagram:

Configuring Ingress features through BackendConfig parameters

The following sections show you how to set your BackendConfig to enable specific Ingress features. Changes to a BackendConfig resource are constantly reconciled, so you do not need to delete and recreate your Ingress to see that BackendConfig changes are reflected.

For information on BackendConfig limitations, see the limitations section.

Important: For GKE versions 1.16.8-gke.3 and higher, we recommend you use the cloud.google.com/v1 API version. If you are using an earlier GKE version, use cloud.google.com/v1beta1.

Backend service timeout

You can use a BackendConfig to set a backend service timeout period in seconds. If you do not specify a value, the default value is 30 seconds.

The following BackendConfig manifest specifies a timeout of 40 seconds:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 40

Cloud CDN

You can enable Cloud CDN using a BackendConfig.

Note: You cannot enable both IAP and Cloud CDN in a BackendConfig. If the BackendConfig doesn't have an iap block, then any existing IAP settings on the backend service are inherited.

The following BackendConfig manifest shows all the fields available when enabling Cloud CDN:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  cdn:
    enabled: cdnEnabled
    cachePolicy:
      includeHost: includeHost
      includeProtocol:includeProtocol
      includeQueryString: includeQueryString
      queryStringBlacklist:queryStringBlacklist
      queryStringWhitelist: queryStringWhitelist

Replace the following:

• cdnEnabled: If set to true, Cloud CDN is enabled for this Ingress backend.
• includeHost: If set to true, requests to different hosts are cached separately.
• includeProtocol: If set to true, HTTP and HTTPS requests are cached separately.
• includeQueryString: If set to true, query string parameters are included in the cache key according to queryStringBlacklist or queryStringWhitelist. If neither is set, the entire query string is included. If set to false, the entire query string is excluded from the cache key.
• queryStringBlacklist: Specify a string array with the names of query string parameters to exclude from cache keys. All other parameters are included. You can specify queryStringBlacklist or queryStringWhitelist, but not both.
• queryStringWhitelist: Specify a string array with the names of query string parameters to include in cache keys. All other parameters are excluded. You can queryStringBlacklist or queryStringWhitelist, but not both.

Expand the following section to see an example that deploys Cloud CDN through Ingress and then validates that application content is being cached.

Cloud CDN example

To use a BackendConfig to configure Cloud CDN, perform the following tasks:

1. Create a dedicated namespace for this example to run in:
   

   kubectl create namespace cdn-how-to
   

2. Create a Deployment file named my-deployment.yaml based on the following Deployment manifest. This manifest declares that you want to run two replicas of the ingress-gce-echo-amd64 web application.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: cdn-how-to
     name: my-deployment
   spec:
     selector:
       matchLabels:
         purpose: demonstrate-cdn
     replicas: 2
     template:
       metadata:
         labels:
           purpose: demonstrate-cdn
       spec:
         containers:
         - name: echo-amd64
           image: gcr.io/google-samples/hello-app-cdn:1.0
   

3. Create the Deployment resource:

   kubectl apply -f my-deployment.yaml
   

4. Create a BackendConfig named my-backendconfig.yaml based on the following BackendConfig manifest. The manifest specifies a Cloud CDN cache policy and declares that Cloud CDN should be enabled:

     apiVersion: cloud.google.com/v1
     kind: BackendConfig
     metadata:
       namespace: cdn-how-to
       name: my-backendconfig
     spec:
       cdn:
         enabled: true
         cachePolicy:
           includeHost: true
           includeProtocol: true
           includeQueryString: false
   

5. Create the BackendConfig resource:

   kubectl apply -f my-backendconfig.yaml
   

6. Create a file named my-service.yaml based on the following Service manifest:

   apiVersion: v1
   kind: Service
   metadata:
     namespace: cdn-how-to
     name: my-service
     labels:
       purpose: demonstrate-cdn
     annotations:
       cloud.google.com/backend-config: '{"ports": {"80":"my-backendconfig"}}'
   spec:
     type: NodePort
     selector:
       purpose: demonstrate-cdn
     ports:
     - port: 80
       protocol: TCP
       targetPort: 8080
   

7. Create the Service resource:

   kubectl apply -f my-service.yaml
   

8. Create a file named my-ingress.yaml based on the following Ingress manifest:

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     namespace: cdn-how-to
     name: my-ingress
     annotations:
       kubernetes.io/ingress.global-static-ip-name: "cdn-how-to-address"
   spec:
     rules:
     - http:
         paths:
         - path: /*
           backend:
             serviceName: my-service
             servicePort: 80
   

9. Create the Ingress resource:

   kubectl apply -f my-ingress.yaml
   

10. Wait approximately ten minutes for the Kubernetes Ingress controller to configure a Google Cloud load balancer, and then retrieve the IP address used by the load balancer's forwarding rules from the Ingress resource.

    kubectl describe my-ingress | grep "Address"
    

    Output:

    Address:          address
    

    address is your Ingress external IP address.

Validate Cloud CDN caching

To validate if your content is being cached, enter this curl command twice:

curl -v address/?cache=true

Replace address with your Ingress external IP address.

The output shows the response headers and body. In the response headers, you can see that the content was cached. The Age header tells you how many seconds the content has been cached:

HTTP/1.1 200 OK
Date: Fri, 25 Jan 2019 02:34:08 GMT
Content-Length: 70
Content-Type: text/plain; charset=utf-8
Via: 1.1 google
Cache-Control: max-age=86400,public
Age: 2716

Hello, world!
Version: 1.0.0
Hostname: my-deployment-7f589cc5bc-l8kr8
...

If you find that your content is not being cached, make sure that your application is properly configured to enable caching of content. For more information, see Cacheable content.

Connection draining timeout

You can configure connection draining timeout using a BackendConfig. Connection draining timeout is the time, in seconds, to wait for connections to drain. For the specified duration of the timeout, existing requests to the removed backend are given time to complete. The load balancer does not send new requests to the removed backend. After the timeout duration is reached, all remaining connections to the backend are closed. The timeout duration can be from 0 to 3600 seconds. The default value is 0, which also disables connection draining.

Note: If you are using Services that are backed by InstanceGroups, setting a high value for drainingTimeoutSec (for example, 3600 seconds) could cause reconciliation errors in the GKE Ingress controller. If you need a high value, consider using Services that are backed by network endpoint groups.

The following BackendConfig manifest specifies a connection draining timeout of 60 seconds:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  connectionDraining:
    drainingTimeoutSec: 60

Custom health check configuration

There are a variety of ways that GKE configures Google Cloud load balancer health checks when deploying through Ingress. To learn more about how GKE Ingress deploys health checks, see Ingress health checks.

One method is to configure health checks explicitly through a BackendConfig. If set, these parameters supersede the Kubernetes readiness probe settings and health check default values.

You can configure multiple GKE Services to reference the same BackendConfig as a reusable template. For healthCheck parameters, a unique Google Cloud health check is created for each backend service corresponding to each GKE Service.

The following BackendConfig manifest shows all the fields available when configuring a BackendConfig health check:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  healthCheck:
    checkIntervalSec: interval
    timeoutSec: timeout
    healthyThreshold: health-threshold
    unhealthyThreshold: unhealthy-threshold
    type: protocol
    requestPath: path
    port: port

Replace the following:

• interval: Specify the check-interval, in seconds, for each health check prober. This is the time from the start of one prober's check to the start of its next check. If you omit this parameter, the Google Cloud default of 5 seconds is used. For additional implementation details, see Multiple probes and frequency.
• timeout: Specify the amount of time that Google Cloud waits for a response to a probe. The value of timeout must be less than or equal to the interval. Units are seconds. Each probe requires an HTTP 200 (OK) response code to be delivered before the probe timeout.
• health-threshold and unhealthy-threshold: Specify the number of sequential connection attempts that must succeed or fail, for at least one prober, in order to change the health state from healthy to unhealthy or vice versa. If you omit one of these parameters, Google Cloud uses the default value of 2.
• protocol: Specify a protocol used by probe systems for health checking. The BackendConfig only supports creating health checks using the HTTP, HTTPS, or HTTP2 protocols. For more information, see Success criteria for HTTP, HTTPS, and HTTP/2. You cannot omit this parameter.
• path: For HTTP, HTTPS, or HTTP2 health checks, specify the request-path to which the probe system should connect. If you omit this parameter, Google Cloud uses the default of /.
• port: Specifies the port by using a port number. If you omit this parameter, Google Cloud uses the default of 80. When using container native load balancing, probe systems connect to a serving Pod's IP address. Otherwise, health check probes connect to the IP address of the Node on which the serving Pod is running.

Google Cloud Armor security policy

Google Cloud Armor security policies help you protect your load-balanced applications from web-based attacks. Once you have configured a Google Cloud Armor security policy, you can reference it using a BackendConfig.

Add the name of your security policy to the BackendConfig. The following BackendConfig manifest specifies a security policy named example-security-policy:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  namespace: cloud-armor-how-to
  name: my-backendconfig
spec:
  securityPolicy:
    name: "example-security-policy"

HTTP access logging

Note: Access logging defaults to "on" in all versions prior to GKE 1.18 but is exposed as a configurable field starting in 1.16.8-gke.10.

Ingress can log all HTTP requests from clients to Cloud Logging. You can enable and disable access logging using BackendConfig along with setting the access logging sampling rate.

To configure access logging, use the logging field in your BackendConfig. If logging is omitted, access logging takes the default behavior. This is dependent on the GKE version.

You can configure the following fields:

• enable: If set to true, access logging will be enabled for this Ingress and logs is available in Cloud Logging. Otherwise, access logging is disabled for this Ingress.
• sampleRate: Specify a value from 0.0 through 1.0, where 0.0 means no packets are logged and 1.0 means 100% of packets are logged. This field is only relevant if enable is set to true. sampleRate is an optional field, but if it's configured then enable: true must also be set or else it is interpreted as enable: false.

The following BackendConfig manifest enables access logging and sets the sample rate to 50% of the HTTP requests for a given Ingress resource:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  logging:
    enable: true
    sampleRate: 0.5

Identity-Aware Proxy

To configure the BackendConfig for Identity-Aware Proxy IAP, you need to specify the enabled and secretName values to your iap block in your BackendConfig. To specify these values, ensure that you have the compute.backendServices.update permission.

Note: You cannot enable both IAP and Cloud CDN in a BackendConfig. If the BackendConfig doesn't have an iap block, then any existing IAP settings on the backend service are inherited.

The following BackendConfig manifest enables Identity-Aware Proxy:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name:  my-backendconfig
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: my-secret

For full instructions, see Enabling IAP for GKE in the IAP documentation.

Session affinity

You can use a BackendConfig to set session affinity to client IP or generated cookie.

Important: Use a VPC-native cluster if you want to configure session affinity. Session affinity is useful only for Services that are backed by network endpoint groups, and network endpoint groups require a VPC-native cluster.

Client IP affinity

To use a BackendConfig to set client IP affinity, set affinityType to "CLIENT_IP", as in the following example:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  sessionAffinity:
    affinityType: "CLIENT_IP"

Generated cookie affinity

To use a BackendConfig to set generated cookie affinity , set affinityType to GENERATED_COOKIE in your BackendConfig manifest. You can also use affinityCookieTtlSec to set the time period for the cookie to remain active.

The following manifest sets the affinity type to generated cookie and gives the cookies a TTL of 50 seconds:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  sessionAffinity:
    affinityType: "GENERATED_COOKIE"
    affinityCookieTtlSec: 50

User-defined request headers

You can use a BackendConfig to configure user-defined request headers. The load balancer adds the headers you specify to the requests it forwards to the backends.

To enable user-defined request headers, you specify a list of headers in the customRequestHeaders property of the BackendConfig resource. Specify each header as a header-name:header-value string.

The following BackendConfig manifest adds three request headers:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  customRequestHeaders:
    headers:
    - "X-Client-Region:{client_region}"
    - "X-Client-City:{client_city}"
    - "X-Client-CityLatLong:{client_city_lat_long}"

Exercise: Setting Ingress timeouts using a backend service

The following exercise shows you the steps required for configuring timeouts and connection draining with an Ingress with a BackendConfig resource.

To configure the backend properties of an Ingress, complete the following tasks:

1. Create a Deployment.
2. Create a BackendConfig.
3. Create a Service, and associate one of its ports with the BackendConfig.
4. Create an Ingress, and associate the Ingress with the (Service, port) pair.
5. Validate the properties of the backend service.
6. Clean up.

Creating a Deployment

Before you create a BackendConfig and a Service, you need to create a Deployment.

The following example manifest is for a Deployment named my-deployment.yaml:

# my-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      purpose: bsc-config-demo
  replicas: 2
  template:
    metadata:
      labels:
        purpose: bsc-config-demo
    spec:
      containers:
      - name: hello-app-container
        image: gcr.io/google-samples/hello-app:1.0

Create the Deployment by running the following command:

kubectl apply -f my-deployment.yaml

Creating a BackendConfig

Use your BackendConfig to specify the Ingress features you want to use.

This BackendConfig manifest, named my-backendconfig.yaml, specifies:

• A timeout of 40 seconds.
• A connection draining timeout of 60 seconds.

# my-backendconfig.yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 40
  connectionDraining:
    drainingTimeoutSec: 60

For the complete list of features you can set using a BackendConfig, refer to the BackendConfig section of the features table.

Create the BackendConfig by running the following command:

kubectl apply -f my-backendconfig.yaml

Creating a Service

A BackendConfig corresponds to a single Service-Port combination, even if a Service has multiple ports. Each port can be associated with a single BackendConfig CRD. If a Service port is referenced by an Ingress, and if the Service port is associated with a BackendConfig, then the HTTP(S) load balancing backend service takes part of its configuration from the BackendConfig.

The following is an example Service manifest named my-service.yaml:

# my-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    purpose: bsc-config-demo
  annotations:
    cloud.google.com/backend-config: '{"ports": {"80":"my-backendconfig"}}'
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: ClusterIP
  selector:
    purpose: bsc-config-demo
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080

The cloud.google.com/backend-config annotation specifies a mapping between ports and BackendConfig objects. In my-service.yaml:

• Any Pod that has the label purpose: bsc-config-demo is a member of the Service.
• TCP port 80 of the Service is associated with a BackendConfig named my-backendconfig. The cloud.google.com/backend-config annotation specifies this.
• A request sent to port 80 of the Service is forwarded to one of the member Pods on port 8080.

To create the Service, run the following command:

kubectl apply -f my-service.yaml

Creating an Ingress

The following is an Ingress manifest named my-ingress.yaml. In this example, incoming requests are routed to port 80 of the Service named my-service.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: my-service
          servicePort: 80

To create the Ingress, run the following command:

kubectl apply -f my-ingress.yaml

Wait a few minutes for the Ingress controller to configure an HTTP(S) Load Balancing and an associated backend service. Once this is complete, you have configured your Ingress to use a timeout of 40 seconds and a connection draining timeout of 60 seconds.

Validating backend service properties

You can validate that the correct load balancer settings have been applied through your BackendConfig. To do this, identify the backend service that Ingress has deployed and inspect its settings to validate that they match the Deployment manifests.

First, describe the my-ingress resource and filter for the annotation that lists the backend services associated with the Ingress. For example:

kubectl describe ingress my-ingress | grep ingress.kubernetes.io/backends

You should see output similar to the following:

ingress.kubernetes.io/backends: '{"k8s1-27fde173-default-my-service-80-8d4ca500":"HEALTHY","k8s1-27fde173-kube-system-default-http-backend-80-18dfe76c":"HEALTHY"}

The output provides information about your backend services. For example, this annotation contains two backend services:

• "k8s1-27fde173-default-my-service-80-8d4ca500":"HEALTHY" provides information about the backend service associated with the my-service Kubernetes Service.
  • k8s1-27fde173 is a hash used to describe the cluster.
  • default is the Kubernetes namespace.
  • HEALTHY indicates that the backend is healthy.
• "k8s1-27fde173-kube-system-default-http-backend-80-18dfe76c":"HEALTHY" provides information about the backend service associated with the default backend (404-server).
  • k8s1-27fde173 is a hash used to describe the cluster.
  • kube-system is the namespace.
  • default-http-backend is the Kubernetes Service name.
  • 80 is the host port.
  • HEALTHY indicates that the backend is healthy.

Next, inspect the backend service associated with my-service using gcloud. Filter for "drainingTimeoutSec" and "timeoutSec" to confirm that they've been set in the Google Cloud Load Balancer control plane. For example:

# Optionally, set a variable
export BES=k8s1-27fde173-default-my-service-80-8d4ca500

# Filter for drainingTimeoutSec and timeoutSec
gcloud compute backend-services describe ${BES} --global | grep -e "drainingTimeoutSec" -e "timeoutSec"

Output:

  drainingTimeoutSec: 60
  timeoutSec: 40

Seeing drainingTimeoutSec and timeoutSec in the output confirms that their values were correctly set through the BackendConfig.

Cleaning up

To prevents unwanted charges incurring on your account, delete the Kubernetes objects that you created for this exercise:

kubectl delete ingress my-ingress
kubectl delete service my-service
kubectl delete backendconfig my-backendconfig
kubectl delete deployment my-deployment

BackendConfig limitations

BackendConfigs have the following limitations:

• Only one (Service, port) pair can consume only one BackendConfig, even if multiple Ingress objects reference the (Service, port). This means all Ingress objects that reference the same (Service, port) must use the same configuration for Google Cloud Armor, IAP, and Cloud CDN.

• IAP and Cloud CDN cannot be enabled for the same HTTP(S) Load Balancing backend service. This means that you cannot configure both IAP and Cloud CDN in the same BackendConfig.

• You must use kubectl 1.7 or later to interact with BackendConfig.

Removing the configuration specified in a FrontendConfig or BackendConfig

To revoke an Ingress feature, you must explicitly disable the feature configuration in the FrontendConfig or BackendConfig CRD. The Ingress controller only reconciles configurations specified in these CRDs.

To clear or disable a previously enabled configuration, set the field's value to an empty string ("") or to a Boolean value of false, depending on the field type.

The following BackendConfig manifest disables a Google Cloud Armor security policy and Cloud CDN:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  cdn:
    enabled: false
  securityPolicy:
    name: ""

Deleting a FrontendConfig or BackendConfig

FrontendConfig

To delete a FrontendConfig, follow these steps:

1. Remove the FrontendConfig's name from the networking.gke.io/v1beta1.FrontendConfig annotation in the Ingress manifest.

2. Apply the changed Ingress manifest to your cluster. For example, use kubectl apply.

3. Delete the FrontendConfig. For example, use kubectl delete frontendconfig config my-frontendconfig.

Caution: Do not delete a FrontendConfig object before you remove its name from the networking.gke.io/v1beta1.FrontendConfig annotation and apply the updated Ingress manifest. If you do, the configuration does not get removed from the HTTP(S) Load Balancing backend service. Also, you will see errors as Kubernetes events on the Ingress object.

BackendConfig

To delete a BackedConfig, follow these steps:

1. Remove the BackendConfig's name from the cloud.google.com/backend-config annotation in the Service manifest.

2. Apply the changed Service manifest to your cluster. For example, use kubectl apply.

3. Delete the BackendConfig. For example, use kubectl delete backendconfig my-backendconfig.

Caution: Do not delete a BackendConfig object before you remove its name from the cloud.google.com/backend-config annotation and apply the updated Service manifest. If you do, the configuration does not get removed from the HTTP(S) Load Balancing backend service. Also, you will see errors as Kubernetes events on the Ingress object.

Troubleshooting

BackendConfig not found

This error occurs when a BackendConfig for a Service port is specified in the Service annotation, but the actual BackendConfig resource could not be found. This can occur if you have not created the BackendConfig resource, created it in the wrong namespace, or misspelled the reference in the Service annotation.

To evaluate a Kubernetes event, run the following command:

kubectl get event

The following type of output indicates your BackendConfig was not found:

KIND    ... SOURCE
Ingress ... loadbalancer-controller

MESSAGE
Error during sync: error getting BackendConfig for port 80 on service "default/my-service":
no BackendConfig for service port exists

Security policy not found

After the Ingress object is created, if the security policy isn't properly associated with the load balancer service, evaluate the Kubernetes event to see if there is a configuration mistake. If your BackendConfig specifies a nonexistent policy, a warning event periodically emitted. To fix this problem, make sure you specify the correct security policy, by name, in your BackendConfig.

To evaluate a Kubernetes event, run the following command:

kubectl get event

The following type of output indicates your security policy was not found:

KIND    ... SOURCE
Ingress ... loadbalancer-controller

MESSAGE
Error during sync: The given security policy "my-policy" does not exist.

What's next

• GKE Ingress for single-cluster load balancing.
• Multi-cluster Ingress load balancing.
• Ingress tutorial for deploying HTTP(S) routing for GKE apps.