L4 load balancer service not accepting connections

Problem

When trying to access the service's exposing load balancer from within the cluster, it fails.

Environment

  • Google Kubernetes Engine private cluster
  • Cloud NAT
  • Service exposed with external Network Load Balancer and configured to allow access only from Cloud NAT IP

Solution

  1. Add pod IP range to the spec:loadBalancerSourceRanges in the service definition.

Cause

All the cluster nodes are added as backend to the External Network Load Balancer. So when the pod communicates with the LB IP, it will not go through Cloud NAT even though it is a public IP. It will be routed/routable by the node. Review Three-tier web service with global access example to get more information about how Google programs backends to respond to L4 Load Balancers.