IP masquerade agent

Autopilot Standard

This page explains how IP masquerading works in Google Kubernetes Engine (GKE) and provides configuration options for different scenarios.

Overview of IP masquerading

IP masquerading is a form of source network address translation (SNAT) used to perform many-to-one IP address translations. GKE can use IP masquerading to change the source IP addresses of packets sent from Pods. When IP masquerading applies to a packet emitted by a Pod, GKE changes the packet's source address from the Pod IP to the underlying node's IP address. Masquerading a packet's source is useful when a recipient is configured to receive packets only from the cluster's node IP addresses.

On Linux nodes, GKE configures iptables rules. GKE uses the ip-masq-agent DaemonSet to configure the appropriate dataplane. IP masquerading is not supported with Windows Server node pools.

For a more general overview of IP masquerading in any Kubernetes implementation, see IP Masquerade Agent User Guide.

IP masquerading in GKE

GKE performs IP masquerading or SNAT under the following conditions:

For clusters in Autopilot mode:

GKE version is 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later.
The cluster was created with GKE Dataplane V2 enabled.
The Egress NAT policy was deployed.

For clusters in Standard mode:

The ip-masq-agent DaemonSet is installed.
The ip-masq-agent ConfigMap includes nonMasqueradeCIDRs and masqLinkLocal parameters.

The following table describes valid masquerading outcomes based on the cluster mode and configuration:

Cluster mode	Cluster configuration	SNAT behavior
Autopilot	Cluster is running the Egress NAT policy, and Cluster version is 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later, and You have created the cluster with GKE Dataplane V2 enabled.	GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. GKE preserves the source Pod IP addresses for packets sent to the internal IP ranges for Pods, Services, and nodes on the cluster. You can identify these destinations with the `gke-{CLUSTER_SHORT_HASH}` hash. For more information, see Egress NAT policy.
Standard	The `ip-masq-agent` is installed in your cluster, and You have specified a `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap	GKE preserves the source Pod IP addresses for packets sent to destinations specified in the `nonMasqueradeCIDRs` list. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in the `nonMasqueradeCIDRs` list.
Standard	The `ip-masq-agent` is not installed in your cluster, or you have not specified a `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap. and You have created the cluster without the `--disable-default-snat` flag.	GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.
Standard	The `ip-masq-agent` is not installed in your cluster, or you have not specified a `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap. and You have created the cluster with the `--disable-default-snat` flag.	GKE preserves the source Pod IP addresses for packets sent to all destinations. Change this behavior by ensuring `ip-masq-agent` is installed and you have specified a `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap.

Masquerading scenarios

The following table summarizes possible SNAT scenarios based on cluster mode, expected behavior, and corresponding GKE cluster configuration:

Cluster mode	Scenario	SNAT behavior	Cluster configuration
Autopilot	Not masquerade to any destinations in Autopilot mode clusters	The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations without affecting the cluster communication between nodes, pods, and services.	Create a cluster in versions 1.23.4-gke.1600 or later or 1.22.7-gke.1500 or later, with GKE Dataplane V2 enabled. Remove the default policy and all your custom created policies. Deploy a new Egress NAT policy and set a CIDR with the `0.0.0.0/0` range. For more information, see how to deploy a new Egress NAT policy.
Autopilot	Not masquerade outbound traffic to on-premises with PUPI (privately used public IP) ranges.	Pod IP is accessible from on-premises networks and has routing and firewall rules set up to allow it. You can preserve the source IP address for traffic coming from pods.	Create a new Egress NAT policy that contains the privately used public IP (PUPI) addresses. For more information, see how to deploy a new Egress NAT policy.
Standard	Masquerade to destinations out of `100.64.0.0/24` and `10.0.0.0/8`	The cluster preserves the source Pod IP addresses (non-masquerading) for packets sent to destinations `100.64.0.0/24` and `10.0.0.0/8`. The cluster changes the source Pod IP addresses (masquerading) to node IP addresses for packets sent to all other destinations.	Deploy the `ip-masq-agent` DaemonSet. Set the `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap to `[100.64.0.0/24,10.0.0.0/8]`. For more information, see how to create the `ip-masq-agent` ConfigMap.
Standard	Masquerade to destinations outside of the default non-masquerade destinations.	The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to the default non-masquerade destinations.	Create the cluster without the `--disable-default-snat` flag. Ensure that either or both of the following are true: The `ip-masq-agent` DaemonSet is not deployed, or The `nonMasqueradeCIDRs` key is absent in the `ip-masq-agent` ConfigMap. For more information, see how to delete the `ip-masq-agent`.
Standard	Not masquerade to any destinations in Standard mode clusters	The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations.	Configure your cluster as one of the following options: Option 1: Create the cluster with the `--disable-default-snat` flag Ensure that either or both of these are true: Ensure the `ip-masq-agent` DaemonSet is not deployed, or Ensure the `nonMasqueradeCIDRs` key is absent in the `ip-masq-agent` ConfigMap. Option 2: Deploy the `ip-masq-agent` DaemonSet. Set the `nonMasqueradeCIDRs` list in the `ip-masq-agent` ConfigMap to `[0.0.0.0/0]`. For more information, see how to configure the `ip-masq-agent`.

Diagnostic containers and Pods with `hostNetwork: true`

IP masquerading is not applicable to packets sent from the following containers. Unless you specify a custom source IP address for packets, these containers send packets with sources set to the node's primary internal IP address:

Diagnostic containers for debugging node issues using toolbox.
Containers in Pods with spec.hostNetwork: true

When is the `ip-masq-agent` installed

The ip-masq-agent DaemonSet is automatically installed on nodes of your cluster if you created the cluster without the --disable-default-snat flag, and one or more of the following conditions are true:

The cluster has network policy enabled.
The cluster's Pod IP address range does not match or is not within 10.0.0.0/8.

For the ip-masq-agent DaemonSet to be effective, you must also specify the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. For more information, see how to configure an IP masquerade agent.

GKE constantly updates and reconciles the ip-masq-agent DaemonSet.

Default non-masquerade destinations

The following table summarizes GKE default non-masquerade destination ranges.

GKE versions	Destination ranges
Versions earlier than 1.14	Varies by version and image type.
Versions equal to or later than: 1.14.1-gke.14 1.14.2-gke.1	`10.0.0.0/8` `172.16.0.0/12` `192.168.0.0/16` `100.64.0.0/10` `192.0.0.0/24` `192.0.2.0/24` `192.88.99.0/24` `198.18.0.0/15` `198.51.100.0/24` `203.0.113.0/24` `240.0.0.0/4`

The default non-masquerade destinations are only relevant when the cluster meets the following conditions:

You created the cluster without the --disable-default-snat flag , and
One or both of the following are true:
- You have not deployed the ip-masq-agent DaemonSet.
- The nonMasqueradeCIDRs list is not specified in the ip-masq-agent ConfigMap. This also includes the case where you don't specify the ip-masq-agent ConfigMap file.

For a cluster meeting the preceding conditions, GKE takes the following actions:

The cluster preserves source Pod IP addresses if the packet's destination is in the list of default non-masquerade destinations.
The cluster changes the source IP addresses from the Pod IP address to the IP address of the node running the Pod, if the packet's destination is not in the list of default non-masquerade destinations.

If you configure the cluster where both the ip-masq-agent DaemonSet is deployed and a nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap, then the default non-masquerade destinations no longer apply. This is because the non-masquerade destinations are specified in the nonMasqueradeCIDRs list instead.

Effect of the `--disable-default-snat` flag

The following describe the purpose of the --disable-default-snat flag:

When you create a cluster with the --disable-default-snat flag, GKE preserves source Pod IP addresses for packets sent to all destinations if one or both of the following are true:
- The ip-masq-agent DaemonSet is not deployed.
- The nonMasqueradeCIDRs list is not specified in the ip-masq-agent ConfigMap (including the case where the ip-masq-agent ConfigMap is not specified at all).
If your cluster meets either or both of these conditions, then GKE preserves source Pod IP addresses for packets sent to any destination.
If your cluster has Pods which use privately used public IP addresses, you must create the cluster with the --disable-default-snat flag.
The --disable-default-snat flag does not override a valid ip-masq-agent DaemonSet and configuration. If you configure the cluster such that both the ip-masq-agent DaemonSet is deployed and a nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap, then the --disable-default-snat flag is no longer relevant. With this DaemonSet and its configuration, non-masquerade destinations are explicitly specified in the nonMasqueradeCIDRs list, so masquerading is implied for packets sent to destinations not included in the nonMasqueradeCIDRs list.

Effect of the Egress NAT policy in Autopilot clusters

The GKE Egress NAT policy lets you configure IP masquerading on Autopilot clusters. You can use the GKE Egress NAT policy custom resource definition (CRD) to change the source IP addresses of packets sent from Pods.

For security or IP address exhaustion reasons, you can masquerade IP addresses from Pod to Node IPs for outbound traffic to on-premises networks. For example, you can use a non-RFC-1918 range for Autopilot clusters and use a RFC-1918 range for Nodes. However, if Pods must communicate with on-premises networks that also use non-RFC-1918 range, IP addresses can overlap. To avoid traffic loss, you can configure an Egress NAT Policy to not advertise the Pods' non-RFC-1918 ranges to on-premises networks. The Egress NAT Policy masquerades the Pods' non RFC-1918 range to use the Node's RFC-1918 range instead. Ensure that a Node range is not overlapping with any on-premises range or it can cause a traffic loop.

GKE enforces the IP masquerading behavior for Autopilot clusters through the following process:

GKE deploys the Egress NAT controller and the ip-masq-agent.
You create the Egress NAT policy.
The GKE controller translates the policy into the ip-masq-agent ConfigMap.
The ip-masq-agent DaemonSet reads the ConfigMap and then GKE enforces the IP masquerading behavior.

Automatically generated policies

GKE supports two automatically generated Egress NAT policies, default and managed by GKE. Default policies are editable. Managed by GKE policies are fixed and are not editable.

Default policy

GKE predefines a set of default IP address ranges. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. To change these default IP address ranges, see Edit and deploy the Egress NAT policy.

The following manifest describes a default Egress NAT policy:

    Name:         default
    Namespace:
    Labels:       <none>
    Annotations:  <none>
    API Version:  networking.gke.io/v1
    Kind:         EgressNATPolicy
    Metadata:
      Creation Timestamp:  2022-03-16T21:05:45Z
      Generation:          2
      Managed Fields:
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            .:
            f:action:
          f:status:
        Manager:      egress-nat-controller
        Operation:    Update
        Time:         2022-03-16T21:05:45Z
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            f:destinations:
        Manager:         kubectl
        Operation:       Update
        Time:            2022-03-17T01:58:13Z
      Resource Version:  189346
      UID:               06acbb5a-23ba-4c2a-bb34-9b6ed8c4a87f
    Spec:
      Action:  NoSNAT
      Destinations:
        Cidr:  10.0.0.0/8
        Cidr:  172.16.0.0/12
        Cidr:  192.168.0.0/16
        Cidr:  240.0.0.0/4
        Cidr:  192.0.2.0/24
        Cidr:  198.51.100.0/24
        Cidr:  203.0.113.0/24
        Cidr:  100.64.0.0/10
        Cidr:  198.18.0.0/15
        Cidr:  192.0.0.0/24
        Cidr:  192.88.99.0/24
    Status:
    Events:  <none>

The CIDR ranges are the same as the default non-masquerade destination ranges.

Managed by GKE policy

GKE Egress NAT policy reserves a static range of IP addresses required to preserve cluster's operation. This static range contains the Pod, Service, and Node IP address ranges of the cluster, and might overlap with the default policy.

You can identify this policy by a dynamic 8-byte hash (gke-{CLUSTER_SHORT_HASH}) that GKE assigns. You cannot edit this policy.

The following manifest describes a managed by GKE policy named gke-bbfa6c0e-1:

    Name:         gke-bbfa6c0e-1
    Namespace:
    Labels:       <none>
    Annotations:  <none>
    API Version:  networking.gke.io/v1
    Kind:         EgressNATPolicy
    Metadata:
      Creation Timestamp:  2022-03-16T21:05:46Z
      Generation:          1
      Managed Fields:
        API Version:  networking.gke.io/v1
        Fields Type:  FieldsV1
        fieldsV1:
          f:spec:
            .:
            f:action:
            f:destinations:
          f:status:
        Manager:         egress-nat-controller
        Operation:       Update
        Time:            2022-03-16T21:05:46Z
      Resource Version:  11699
      UID:               0201b5de-a6f6-4926-822b-31ed7cdee2c6
    Spec:
      Action:  NoSNAT
      Destinations:
        Cidr:  10.119.128.0/17
        Cidr:  10.120.0.0/22
        Cidr:  10.128.0.0/20
    Status:
    Events:  <none>

Masquerading to link-local destinations

The link-local IP address destinations fit in the 169.254.0.0/16 IP range. By default, GKE preserves source Pod IP addresses for packets sent to 169.254.0.0/16. This default configuration applies to all GKE versions regardless of whether the following conditions are met:

The ip-masq-agent DaemonSet is deployed.
The nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap.
The --disable-default-snat flag is specified when creating the cluster.

You can configure GKE to change source Pod IP addresses to node IP addresses when sending packets to 169.254.0.0/16 if you deploy the ip-masq-agent DaemonSet and set masqLinkLocal to True in the ip-masq-agent ConfigMap. When masqLinkLocal is not set, GKE assumes masqLinkLocal is False.