SNAT and NAT in Google Kubernetes Engine clusters

Problem

SNAT and Cloud NAT configurations for Google Kubernetes Engine clusters

Environment

  • Non-private (public), VPC-native Google Kubernetes Engine cluster
  • ip-masq-agent installed
  • nonMasqueradeCIDRs list not specified in the ip-masq-agent's ConfigMap
  • Destination IP address is not part of the default non-masquerade destinations.

Solution

Specify a nonMasqueradeCIDRs list in the ip-masq-agent's ConfigMap, including all destinations where the Cloud NAT IP should be used instead of the node's external IP. Example:

config:

    nonMasqueradeCIDRs:

        - 35.100.0.0/16

If the Cloud NAT IP should be used for all destinations, then the nonMasqueradeCIDRs list can be as follows. Also, in this case, the public nature of the cluster is not used so using a private cluster may be more appropriate.

config:

    nonMasqueradeCIDRs:

        - 0.0.0.0/0

Ensure all IP address ranges are specified in list form in the nonMasqueradeCIDRs list. 

Cause

The issue is caused due to the absence of the nonMasqueradeCIDRs list from the ip-masq-agent's ConfigMap. The existence of the ip-masq-agent alone will result in the Default SNAT of the cluster to be used, which in turn masquerades the source IP of packets egressing pods for all destinations outside the default non-masquerade destinations. Since the destination IP is not part of these addresses, the node's external IP is used. 

Find full context of SNAT and Cloud NAT configurations for Google Kubernetes Engine clusters below.

Terminology
  • Masquerading in this context means changing the source IP of a packet leaving a pod from the pod IP to the node internal IP. 
  • Non-masquerading in this context means preserving the source IP of a packet leaving a pod as the pod IP. 
Private Google Kubernetes Engine Clusters

In a private Google Kubernetes Engine cluster, there are:

  • Pods with internal IPs only.
  • Nodes with internal IPs only. 

Google Kubernetes Engine clusters have a Default SNAT feature. This feature can be enabled during or after cluster creation:

  • in the Console, by selecting to enable it.
  • in gcloud, by creating or updating the cluster without the --disable-default-snat flag.

When the Default SNAT is enabled, the source IP of packets leaving the pods is preserved as the pod IP for packets sent only to default non-masquerade destinations. The source IP is masqueraded as the node internal IP for all other destinations.

The default non-masquarade destinations can be overridden by installing an ip-masq-agent in the cluster and specifying a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. In that case, the Default SNAT is overridden completely and enabling or disabling it is irrelevant.

In this case, the source IP of packets leaving the pod will be:

  • Preserved as the pod IP for destinations in the nonMasqueradeCIDRs list.
  • Changed to the node internal IP for all destinations outside the nonMasqueradeCIDRs list.

SNAT options

To summarize, there are 3 options:

  1. No SNAT: Preserves the pod IP as the source IP for packets egressing pods to all destinations.
  2. Default SNAT: Preserves the pod IP as the source IP for packets egressing pods to a list of default destinations. To all other destinations, the source IP of packets egressing pods will be the node internal IP. 
  3. Custom SNAT: Preserves the pod IP as the source IP for packets egressing pods to a custom list of destinations defined in the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. To all other destinations, the source IP of packets egressing pods will be the node internal IP.
     

Public destinations

Private clusters have only internal IPs both for pods and nodes. When the destinations are public, Cloud NAT is required. If the destination is public, regardless of whether the source IP is masqueraded or not, the NAT IP is used as source IP of the packet, as both the node IP and the pod IP addresses are internal.

Non-private Google Kubernetes Engine Clusters

In a non-private Google Kubernetes Engine cluster, there are:

  • Pods with internal IPs only.
  • Nodes with internal and external IPs.

Google Kubernetes Engine clusters have a Default SNAT feature. This feature is always enabled in non-private clusters. 

The default behavior is that the source IP of packets leaving the pods is preserved as the pod IP for packets sent only to default non-masquerade destinations. The source IP is masqueraded as the node internal IP for all other destinations.

The default non-masquarade destinations can be overridden by installing an ip-masq-agent in the cluster and specifying a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. In that case, the Default SNAT is overridden completely and enabling or disabling it is irrelevant.

In this case, the source IP of packets leaving the pod will be:

  • Preserved as the pod IP for destinations in the nonMasqueradeCIDRs list.
  • Changed to the node internal IP for all destinations outside the nonMasqueradeCIDRs list.
     

SNAT options

To summarise, there are 2 options: 

  1. Default SNAT: Preserves the pod IP as the source IP for packets egressing pods to a list of default destinations. To all other destinations, the source IP of packets egressing pods will be the node internal IP. 

  2. Custom SNAT: Preserves the pod IP as the source IP for packets egressing pods to a custom list of destinations defined in the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. To all other destinations, the source IP of packets egressing pods will be the node internal IP. 
     

Public destinations

What happens when the destinations are public? Non-private clusters' nodes have external IPs, which are always used for packets:

  • Egressing nodes towards public destinations.
  • Egressing pods towards public destinations, when SNAT is used (the packet's source IP is masqueraded as the node IP).

Cloud NAT can only be used when the pod IP is preserved as the source IP of the packet. Additionally, Cloud NAT needs to serve the pod address range.