Upgrade notice for LoadBalancers from 1.7.0 to 1.7.2

Beginning with Kubernetes version 1.7.0, any new Services you create with type LoadBalancer have health checks for nodes enabled by default (provided all nodes in the cluster are running version v1.7.0 or later).

However, there is a known issue in Kubernetes versions 1.7.0 and 1.7.1 that causes nodes to incorrectly respond to health checks from the GCP Network Load Balancer. When this occurs, the network load balancer configured by Kubernetes shows consistently failing health checks across all the cluster's nodes, but traffic is still forwarded to backends.

Although this issue is fixed in version 1.7.2, you might need to take action when upgrading an existing cluster from version 1.7.0 or 1.7.1 to 1.7.2. The upgrade can cause a potential load imbalance. When health checks are functioning correctly for nodes running version 1.7.2, the GCP load balancer forwards all traffic to those nodes and away from version 1.7.0 or 1.7.1 nodes with failing health checks. This imbalance may cause service interruptions if there aren't enough "healthy" nodes (as in, nodes running version 1.7.2) to handle your cluster's traffic load.

You can mitigate this issue by manually removing the health checks on any affected load balancers prior to upgrading the nodes to version 1.7.2.

Determining whether your cluster is affected

If your cluster nodes are running Kubernetes version 1.7.0 or 1.7.1, your cluster might be affected if you have done any of the following:

  • You've created a new Service with --type LoadBalancer.
  • You've updated an existing Service with --type LoadBalancer to another type (ClusterIP, ExternalName, etc.) and later reverted it LoadBalancer.
  • You've updated the sessionAffinity field on an existing LoadBalancer service.
  • You've set the externalTrafficPolicy field to Cluster on an existing LoadBalancer service.

To confirm whether your cluster is affected:

  1. In the Google Cloud Platform Console, navigate to Kubernetes Engine and select your cluster.
  2. Click the Discovery and Load Balancing tab.
  3. Look for a service where the Type field is LoadBalancer and click the service name.
  4. On the Service Details pane, find the LoadBalancer field. This is the GCP LoadBalancer resource attached to your cluster.
  5. Click the Load Balancer name.
  6. You should see the Load Balancing pane; look for the attached health checks. If there is an attached health check named k8s-XXX-node, your cluster is affected.

Repeat the above steps for all load balancer services on your cluster.

You can determine the corresponding Kubernetes Service by using the Advanced Menu as follows:

  1. Click the Forwarding rules tab.
  2. Select your Load Balancer entry.
  3. You should see the service name in the Description field with the following format: {"kubernetes.io/service-name":"$NAMESPACE/$SERVICE_NAME"}.

Upgrade risk mitigation

To mitigate the risk of a potential load imbalance and ensure a safe upgrade to version 1.7.2, do the following:

  1. Prior to upgrade, manually remove the health checks from all affected load balancers in your cluster.
  2. Upgrade your nodes to version 1.7.2.
  3. After upgrading, replace the health checks on each load balancer in your cluster.

Prior to upgrade

To remove the node health check from a load balancer:

  1. In the Google Cloud Platform Console, navigate to Kubernetes Engine and select your cluster.
  2. Click the Discovery and Load Balancing tab.
  3. Locate your affected LoadBalancer service and click the service name.
  4. On the Service Details pane, find the LoadBalancer field. This is the GCP LoadBalancer resource attached to your cluster.
  5. Click the Load Balancer name.
  6. On the Load Balancing pane, click the Advanced Menu link.
  7. In the Advanced Menu, click Target pools.
    1. Note the health check name, k8s-XXX-node, where XXX is the hash ID of your cluster. You'll need this later to restore the health check, if you so desire.
  8. Edit the corresponding target pool for the load balancer to "no health check." This is an in-place update that should not cause downtime on your service.

Repeat the above steps for each affected Load Balancer on your cluster.

Once you've removed the health checks, you can safely upgrade your nodes to version 1.7.2 without the risk of a traffic imbalance to your nodes. The GCP load balancer will continue to forward traffic to nodes without health checks regardless of status; this behavior is identical to that of clusters running Kubernetes version 1.6.x and earlier.

After upgrading

After you've upgraded, you can replace the node health checks on your cluster's load balancers as follows:

  1. In the Google Cloud Platform Console, navigate to your cluster.
  2. Click the Discovery and Load Balancing tab.
  3. Locate your affected LoadBalancer service and click the service name.
  4. On the Service Details pane, find the LoadBalancer field; this is the GCP LoadBalancer resource attached to your cluster.
  5. Click the Load Balancer name.
  6. On the Load Balancing pane, click the Advanced Menu link.
  7. In the Advanced Menu, click Target pools.
  8. Edit the corresponding target pool for the load balancer to the same value of format k8s-XXX-node that you noted earlier when removing the health checks prior to upgrade. This is an in-place update that should not cause downtime on your service.

Repeat the above steps for each affected load balancer on your cluster.

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine