IP masquerade agent

This page explains how IP masquerading works in Google Kubernetes Engine (GKE) and provides configuration options for different scenarios.

Overview of IP masquerading

IP masquerading is a form of source network address translation (SNAT) that performs many-to-one IP address translations. GKE can use IP masquerading to change the source IP addresses of packets sent from Pods.

For a more general overview of IP masquerading in any Kubernetes implementation, see IP Masquerade Agent User Guide.

GKE IP masquerading

When IP masquerading applies to a packet emitted by a Pod, GKE changes the packet's source IP address from the Pod IP address to the underlying node's IP address. Masquerading a packet's source IP address is useful when a recipient is configured to receive packets only from the cluster's node IP addresses.

On Linux nodes, GKE configures iptables rules. GKE uses the ip-masq-agent DaemonSet to configure the appropriate dataplane.

IP masquerading is not supported with Windows Server node pools.

IP masquerading for Standard clusters

In Standard clusters, the cluster's IP masquerade behavior is governed by three factors:

The following table summarizes IP masquerade configurations for Standard GKE clusters:

Cluster configuration Resulting SNAT behavior

The ip-masq-agent DaemonSet is present in the cluster, and a custom nonMasqueradeCIDRs list exists in the ip-masq-agent ConfigMap.

GKE preserves the source Pod IP addresses for packets sent to destinations specified in the nonMasqueradeCIDRs list.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in the nonMasqueradeCIDRs list.

The ip-masq-agent DaemonSet is present in the cluster, but a custom nonMasqueradeCIDRs list does not exist in the ip-masq-agent ConfigMap or the ip-masq-agent ConfigMap is not present at all.

GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.

The ip-masq-agent DaemonSet is not present in the cluster, and you have created the cluster without the --disable-default-snat flag.

GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.

The ip-masq-agent DaemonSet is not present in the cluster, and you have created the cluster with the --disable-default-snat flag.

GKE preserves the source Pod IP addresses for packets sent to all destinations.

Review Preserving Pod IPv4 address sources to internet destinations for important routing considerations when you preserve Pod IPv4 source addresses and need to route packets to the internet.

IP masquerading for Autopilot clusters

In Autopilot clusters, GKE always deploys an ip-masq-agent DaemonSet. Except for packets sent by Pods to the cluster's node, Pod, or Services ranges, you can control the IP masquerade behavior using an EgressNATPolicy. To use an EgressNATPolicy, your Autopilot cluster must meet both of these requirements:

  • The cluster must use GKE version 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later.
  • The cluster must have been created with GKE Dataplane V2 enabled.

The following table summarizes IP masquerade configurations for Autopilot GKE clusters:

Autopilot cluster configuration Resulting SNAT behavior

The cluster includes a custom EgressNATPolicy whose spec.action is NoSNAT containing non-masquerade destinations specified in spec.destinations[].

GKE preserves the source Pod IP addresses for packets sent to destinations specified in spec.destinations[] of the EgressNATPolicy. GKE accomplishes this by translating spec.destinations[] into a nonMasqueradeCIDRs list in an ip-masq-agent configMap.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in spec.destinations[] of the EgressNATPolicy.

The cluster does not include a custom EgressNATPolicy.

The default EgressNATPolicy and Managed by GKE policy both apply, resulting in the following behavior:

  • GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations.
  • GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.

Configuration examples

Expand the following sections to see IP masquerading and configuration examples based on cluster type.

Advanced configuration reference

When the ip-masq-agent is deployed automatically

In Autopilot mode clusters, GKE always deploys an ip-masq-agent DaemonSet.

In Standard clusters, GKE deploys an ip-masq-agent DaemonSet when the --disable-default-snat flag is not set and the cluster uses one of the following configuration combinations:

  • The cluster does not use GKE Dataplane V2, and network policy enforcement is enabled.

  • The cluster uses a Pod IP address range that does not fit within 10.0.0.0/8.

For the ip-masq-agent DaemonSet to be effective, you must also specify the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. For more information, see how to configure an IP masquerade agent.

When a ip-masq-agent DaemonSet is present in a cluster, GKE updates and reconciles a serving Pod on each node of the cluster.

Default non-masquerade destinations

The default non-masquerade destination are:

  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16
  • 100.64.0.0/10
  • 192.0.0.0/24
  • 192.0.2.0/24
  • 192.88.99.0/24
  • 198.18.0.0/15
  • 198.51.100.0/24
  • 203.0.113.0/24
  • 240.0.0.0/4

The default non-masquerade destinations are applicable to clusters with the following configurations:

The default non-masquerade destinations are not applicable to clusters with the following configurations:

Effect of the --disable-default-snat flag

The --disable-default-snat flag changes the default GKE SNAT behavior so that source Pod IP addresses are preserved for packets sent to all destinations. GKE implements the default SNAT behavior by not deploying any ip-masq-agent DaemonSet in the cluster.

The --disable-default-snat flag has no effect when a cluster includes an ip-masq-agent DaemonSet:

  • Because Autopilot clusters always include an ip-masq-agent DaemonSet, the --disable-default-snat flag has no effect on Autopilot clusters.

You can set the --disable-default-snat flag by updating a cluster after it has been created. Provided that the cluster has no ip-masq-agent DaemonSet deployed, disabling default SNAT takes effect after the cluster has replaced all of its nodes—sometimes hours later. This is because GKE respects your configured maintenance windows when it replaces nodes in the cluster. If you haven't configured any maintenance window, you need to manually cycle the nodes in the cluster before the --disable-default-snat flag has any effect.

Link-local masquerading

The 169.254.0.0/16 range is used for link-local IP addresses. Link-local masquerading refers to changing a source Pod IP address to a source node IP address for packets sent to the 169.254.0.0/16 destinations.

Autopilot clusters always preserve source Pod IP addresses for packets sent to 169.254.0.0/16 destinations.

By default, Standard clusters also preserve the source Pod IP addresses for packets sent to 169.254.0.0/16 destinations.

You can enable link-local IP masquerading in a Standard cluster by doing both of the following:

Diagnostic containers and Pods with hostNetwork: true

Unless you specify a custom source IP address for packets, Pods running with hostNetwork: true and diagnostic containers send packets with sources matching the node's IP address. For Pods running with hostNetwork: true, GKE assigns the Pod the node's IP address. GKE does not manage IP addresses for diagnostic containers, including containers for debugging node issues using toolbox.

Autopilot clusters do not support running Pods with spec.hostNetwork: true. Because the nodes of an Autopilot cluster are not accessible using SSH, you cannot run diagnostic containers on them.

Preserving Pod IPv4 address sources to internet destinations

If your cluster's IP masquerade configuration is one of the following, GKE preserves Pod IP address sources for packets sent to all destinations, including internet destinations:

  • In Standard clusters with an ip-masq-agent DaemonSet, if you've set the nonMasqueradeCIDRs to 0.0.0.0 in the ip-masq-agent ConfigMap.
  • In Standard clusters without an ip-masq-agent DaemonSet, if you've set the --disable-default-snat flag.

Pod IPv4 sources are internal IPv4 addresses, which means that they aren't routable on the internet. Consequently, when you preserve source Pod IPv4 addresses for packets sent to the internet, you need to use a technique like one of the following in order to route packets after they leave the nodes of the cluster:

  • Make sure your VPC network has a default route with the default internet gateway next hop, and configure a Cloud NAT gateway to provide public NAT services to at least the subnet secondary IPv4 address ranges used by Pods in your cluster. For more details, see GKE interaction in the Cloud NAT overview.
  • Configure your VPC network to use a custom default route whose next hop is a VM instance or internal passthrough Network Load Balancer, where the VM or backends of the load balancer have been configured to route packets to the internet on behalf of the Pods.

Restoring to the default SNAT behavior

To restore to the default SNAT behavior when an ip-masq-agent DaemonSet is present in a cluster, delete the associated ip-masq-agent ConfigMap. The ip-masq-agent DaemonSet restores the default IP masquerading behavior on the nodes it manages.

To restore to the default SNAT behavior when an ip-masq-agent DaemonSet is not present in a cluster, you will need to upgrade the node pool (make sure --disable-default-snat is not set on the cluster).

Effect of the Egress NAT policy in Autopilot clusters

The GKE Egress NAT policy lets you configure IP masquerading on Autopilot clusters. You can use the GKE Egress NAT policy custom resource definition (CRD) to change the source IP addresses of packets sent from the Pods.

For security or IP address exhaustion reasons, you can masquerade IP addresses from Pod to Node IP address range for outbound traffic to on-premises networks. For example, you can use a non-RFC-1918 range for Autopilot clusters and use a RFC-1918 range for the nodes. However, if Pods must communicate with on-premises networks that also use non-RFC-1918 range, IP addresses can overlap. To avoid traffic loss, you can configure an Egress NAT Policy to not advertise the Pods' non-RFC-1918 ranges to on-premises networks. The Egress NAT Policy masquerades the Pods' non RFC-1918 range to use the Node's RFC-1918 range instead. Ensure that a Node range is not overlapping with any on-premises range or it can cause a traffic loop.

GKE enforces the IP masquerading behavior for Autopilot clusters through the following process:

  1. GKE deploys the Egress NAT controller and the ip-masq-agent.
  2. You create the Egress NAT policy.
  3. The GKE controller translates the policy into the ip-masq-agent ConfigMap.
  4. The ip-masq-agent DaemonSet reads the ConfigMap and then GKE enforces the IP masquerading behavior.

Automatically generated policies

GKE supports the following two automatically generated Egress NAT policies:

  • Default: These policies are editable.
  • Managed by GKE: These policies are fixed and are not editable.

Default policy

GKE predefines a set of default IP address ranges. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. To change these default IP address ranges, see Edit and deploy the Egress NAT policy.

The following manifest describes a default Egress NAT policy:

    Name:         default
    Namespace:
    Labels:       <none>
    Annotations:  <none>
    API Version:  networking.gke.io/v1
    Kind:         EgressNATPolicy
    Metadata:
      Creation Timestamp:  2022-03-16T21:05:45Z
      Generation:          2
      Managed Fields:
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            .:
            f:action:
          f:status:
        Manager:      egress-nat-controller
        Operation:    Update
        Time:         2022-03-16T21:05:45Z
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            f:destinations:
        Manager:         kubectl
        Operation:       Update
        Time:            2022-03-17T01:58:13Z
      Resource Version:  189346
      UID:               06acbb5a-23ba-4c2a-bb34-9b6ed8c4a87f
    Spec:
      Action:  NoSNAT
      Destinations:
        Cidr:  10.0.0.0/8
        Cidr:  172.16.0.0/12
        Cidr:  192.168.0.0/16
        Cidr:  240.0.0.0/4
        Cidr:  192.0.2.0/24
        Cidr:  198.51.100.0/24
        Cidr:  203.0.113.0/24
        Cidr:  100.64.0.0/10
        Cidr:  198.18.0.0/15
        Cidr:  192.0.0.0/24
        Cidr:  192.88.99.0/24
    Status:
    Events:  <none>

The CIDR ranges are the same as the default non-masquerade destination ranges.

Managed by GKE policy

GKE Egress NAT policy reserves a static range of IP addresses required to preserve cluster's operation. This static range contains the Pod, Service, and Node IP address ranges of the cluster, and might overlap with the default policy.

You can identify this policy by a dynamic 8-byte hash (gke-{CLUSTER_SHORT_HASH}) that GKE assigns. You cannot edit this policy.

The following manifest describes a managed by GKE policy named gke-bbfa6c0e-1:

    Name:         gke-bbfa6c0e-1
    Namespace:
    Labels:       <none>
    Annotations:  <none>
    API Version:  networking.gke.io/v1
    Kind:         EgressNATPolicy
    Metadata:
      Creation Timestamp:  2022-03-16T21:05:46Z
      Generation:          1
      Managed Fields:
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            .:
            f:action:
            f:destinations:
          f:status:
        Manager:         egress-nat-controller
        Operation:       Update
        Time:            2022-03-16T21:05:46Z
      Resource Version:  11699
      UID:               0201b5de-a6f6-4926-822b-31ed7cdee2c6
    Spec:
      Action:  NoSNAT
      Destinations:
        Cidr:  10.119.128.0/17
        Cidr:  10.120.0.0/22
        Cidr:  10.128.0.0/20
    Status:
    Events:  <none>

What's next