LoadBalancer Services with Local externalTrafficPolicy cause downtime on backend Pod termination

Problem

Google Kubernetes Engine LoadBalancer with externalTrafficPolicy set to Local produces downtime when pods configured as backends are terminated for a few seconds.

Environment

  • Google Kubernetes Engine on version prior to 1.26

Solution

You must upgrade Google Kubernetes Engine control plane version to 1.26 or later.

Cause

It's expected behavior in k8s earlier than version 1.26. There's an OSS feature called ProxyTerminatingEndpoints which provides graceful termination for externalTrafficPolicy=Local services, but that only is enabled by default in k8s and Google Kubernetes Engine 1.26.