IP masquerade agent


This page explains how IP masquerading works in Google Kubernetes Engine (GKE) and provides configuration options for different scenarios.

Overview of IP masquerading

IP masquerading is a form of source network address translation (SNAT) used to perform many-to-one IP address translations. GKE can use IP masquerading to change the source IP addresses of packets sent from Pods. When IP masquerading applies to a packet emitted by a Pod, GKE changes the packet's source address from the Pod IP to the underlying node's IP address. Masquerading a packet's source is useful when a recipient is configured to receive packets only from the cluster's node IP addresses.

On Linux nodes, GKE configures iptables rules. GKE uses the ip-masq-agent DaemonSet to configure the appropriate dataplane. IP masquerading is not supported with Windows Server node pools.

For a more general overview of IP masquerading in any Kubernetes implementation, see IP Masquerade Agent User Guide.

IP masquerading in GKE

GKE performs IP masquerading or SNAT under the following conditions:

For clusters in Autopilot mode:

  • GKE version is 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later.
  • The cluster was created with GKE Dataplane V2 enabled.
  • The Egress NAT policy was deployed.

For clusters in Standard mode:

The following table describes valid masquerading outcomes based on the cluster mode and configuration:

Cluster mode Cluster configuration SNAT behavior
Autopilot
  • Cluster is running the Egress NAT policy, and
  • Cluster version is 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later, and
  • You have created the cluster with GKE Dataplane V2 enabled.

GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations.

GKE preserves the source Pod IP addresses for packets sent to the internal IP ranges for Pods, Services, and nodes on the cluster. You can identify these destinations with the gke-{CLUSTER_SHORT_HASH} hash.

For more information, see Egress NAT policy.

Standard

GKE preserves the source Pod IP addresses for packets sent to destinations specified in the nonMasqueradeCIDRs list.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in the nonMasqueradeCIDRs list.

Standard

GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.

Standard

GKE preserves the source Pod IP addresses for packets sent to all destinations.

Change this behavior by ensuring ip-masq-agent is installed and you have specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap.

Masquerading scenarios

The following table summarizes possible SNAT scenarios based on cluster mode, expected behavior, and corresponding GKE cluster configuration:

Cluster mode Scenario SNAT behavior Cluster configuration
Autopilot Not masquerade to any destinations in Autopilot mode clusters

The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations without affecting the cluster communication between nodes, pods, and services.

  • Create a cluster in versions 1.23.4-gke.1600 or later or 1.22.7-gke.1500 or later, with GKE Dataplane V2 enabled.
  • Remove the default policy and all your custom created policies.
  • Deploy a new Egress NAT policy and set a CIDR with the 0.0.0.0/0 range.
  • For more information, see how to deploy a new Egress NAT policy.

    Autopilot Not masquerade outbound traffic to on-premises with PUPI (privately used public IP) ranges.

    Pod IP is accessible from on-premises networks and has routing and firewall rules set up to allow it. You can preserve the source IP address for traffic coming from pods.

    Create a new Egress NAT policy that contains the privately used public IP (PUPI) addresses.

    For more information, see how to deploy a new Egress NAT policy.

    Standard Masquerade to destinations out of 100.64.0.0/24 and 10.0.0.0/8
  • The cluster preserves the source Pod IP addresses (non-masquerading) for packets sent to destinations 100.64.0.0/24 and 10.0.0.0/8.
  • The cluster changes the source Pod IP addresses (masquerading) to node IP addresses for packets sent to all other destinations.
  • Deploy the ip-masq-agent DaemonSet.
  • Set the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap to [100.64.0.0/24,10.0.0.0/8].
  • For more information, see how to create the ip-masq-agent ConfigMap.

    Standard Masquerade to destinations outside of the default non-masquerade destinations.

    The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to the default non-masquerade destinations.

  • Create the cluster without the --disable-default-snat flag.
  • Ensure that either or both of the following are true:
    • The ip-masq-agent DaemonSet is not deployed, or
    • The nonMasqueradeCIDRs key is absent in the ip-masq-agent ConfigMap.
  • For more information, see how to delete the ip-masq-agent.

    Standard Not masquerade to any destinations in Standard mode clusters

    The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations.

    Configure your cluster as one of the following options:
  • Option 1:
    • Create the cluster with the --disable-default-snat flag
    • Ensure that either or both of these are true:
      • Ensure the ip-masq-agent DaemonSet is not deployed, or
      • Ensure the nonMasqueradeCIDRs key is absent in the ip-masq-agent ConfigMap.
  • Option 2:
    • Deploy the ip-masq-agent DaemonSet.
    • Set the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap to [0.0.0.0/0].
  • For more information, see how to configure the ip-masq-agent.

    Diagnostic containers and Pods with hostNetwork: true

    IP masquerading is not applicable to packets sent from the following containers. Unless you specify a custom source IP address for packets, these containers send packets with sources set to the node's primary internal IP address:

    When is the ip-masq-agent installed

    The ip-masq-agent DaemonSet is automatically installed on nodes of your cluster if you created the cluster without the --disable-default-snat flag, and one or more of the following conditions are true:

    • The cluster has network policy enabled.
    • The cluster's Pod IP address range does not match or is not within 10.0.0.0/8.

    For the ip-masq-agent DaemonSet to be effective, you must also specify the nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap. For more information, see how to configure an IP masquerade agent.

    GKE constantly updates and reconciles the ip-masq-agent DaemonSet.

    Default non-masquerade destinations

    The following table summarizes GKE default non-masquerade destination ranges.

    GKE versions Destination ranges
    Versions earlier than 1.14 Varies by version and image type.
    Versions equal to or later than:
  • 1.14.1-gke.14
  • 1.14.2-gke.1
  • 10.0.0.0/8
    172.16.0.0/12
    192.168.0.0/16
    100.64.0.0/10
    192.0.0.0/24
    192.0.2.0/24
    192.88.99.0/24
    198.18.0.0/15
    198.51.100.0/24
    203.0.113.0/24
    240.0.0.0/4

    The default non-masquerade destinations are only relevant when the cluster meets the following conditions:

    For a cluster meeting the preceding conditions, GKE takes the following actions:

    • The cluster preserves source Pod IP addresses if the packet's destination is in the list of default non-masquerade destinations.
    • The cluster changes the source IP addresses from the Pod IP address to the IP address of the node running the Pod, if the packet's destination is not in the list of default non-masquerade destinations.

    If you configure the cluster where both the ip-masq-agent DaemonSet is deployed and a nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap, then the default non-masquerade destinations no longer apply. This is because the non-masquerade destinations are specified in the nonMasqueradeCIDRs list instead.

    Effect of the --disable-default-snat flag

    The following describe the purpose of the --disable-default-snat flag:

    • When you create a cluster with the --disable-default-snat flag, GKE preserves source Pod IP addresses for packets sent to all destinations if one or both of the following are true:

      • The ip-masq-agent DaemonSet is not deployed.
      • The nonMasqueradeCIDRs list is not specified in the ip-masq-agent ConfigMap (including the case where the ip-masq-agent ConfigMap is not specified at all).

      If your cluster meets either or both of these conditions, then GKE preserves source Pod IP addresses for packets sent to any destination.

    • If your cluster has Pods which use privately used public IP addresses, you must create the cluster with the --disable-default-snat flag.

    • The --disable-default-snat flag does not override a valid ip-masq-agent DaemonSet and configuration. If you configure the cluster such that both the ip-masq-agent DaemonSet is deployed and a nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap, then the --disable-default-snat flag is no longer relevant. With this DaemonSet and its configuration, non-masquerade destinations are explicitly specified in the nonMasqueradeCIDRs list, so masquerading is implied for packets sent to destinations not included in the nonMasqueradeCIDRs list.

    Effect of the Egress NAT policy in Autopilot clusters

    The GKE Egress NAT policy lets you configure IP masquerading on Autopilot clusters. You can use the GKE Egress NAT policy custom resource definition (CRD) to change the source IP addresses of packets sent from Pods.

    For security or IP address exhaustion reasons, you can masquerade IP addresses from Pod to Node IPs for outbound traffic to on-premises networks. For example, you can use a non-RFC-1918 range for Autopilot clusters and use a RFC-1918 range for Nodes. However, if Pods must communicate with on-premises networks that also use non-RFC-1918 range, IP addresses can overlap. To avoid traffic loss, you can configure an Egress NAT Policy to not advertise the Pods' non-RFC-1918 ranges to on-premises networks. The Egress NAT Policy masquerades the Pods' non RFC-1918 range to use the Node's RFC-1918 range instead. Ensure that a Node range is not overlapping with any on-premises range or it can cause a traffic loop.

    GKE enforces the IP masquerading behavior for Autopilot clusters through the following process:

    1. GKE deploys the Egress NAT controller and the ip-masq-agent.
    2. You create the Egress NAT policy.
    3. The GKE controller translates the policy into the ip-masq-agent ConfigMap.
    4. The ip-masq-agent DaemonSet reads the ConfigMap and then GKE enforces the IP masquerading behavior.

    Automatically generated policies

    GKE supports two automatically generated Egress NAT policies, default and managed by GKE. Default policies are editable. Managed by GKE policies are fixed and are not editable.

    Default policy

    GKE predefines a set of default IP address ranges. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. To change these default IP address ranges, see Edit and deploy the Egress NAT policy.

    The following manifest describes a default Egress NAT policy:

        Name:         default
        Namespace:
        Labels:       <none>
        Annotations:  <none>
        API Version:  networking.gke.io/v1
        Kind:         EgressNATPolicy
        Metadata:
          Creation Timestamp:  2022-03-16T21:05:45Z
          Generation:          2
          Managed Fields:
            API Version:  networking.gke.io/v1
            Fields Type:  FieldsV1
            fieldsV1:
              f:spec:
                .:
                f:action:
              f:status:
            Manager:      egress-nat-controller
            Operation:    Update
            Time:         2022-03-16T21:05:45Z
            API Version:  networking.gke.io/v1
            Fields Type:  FieldsV1
            fieldsV1:
              f:spec:
                f:destinations:
            Manager:         kubectl
            Operation:       Update
            Time:            2022-03-17T01:58:13Z
          Resource Version:  189346
          UID:               06acbb5a-23ba-4c2a-bb34-9b6ed8c4a87f
        Spec:
          Action:  NoSNAT
          Destinations:
            Cidr:  10.0.0.0/8
            Cidr:  172.16.0.0/12
            Cidr:  192.168.0.0/16
            Cidr:  240.0.0.0/4
            Cidr:  192.0.2.0/24
            Cidr:  198.51.100.0/24
            Cidr:  203.0.113.0/24
            Cidr:  100.64.0.0/10
            Cidr:  198.18.0.0/15
            Cidr:  192.0.0.0/24
            Cidr:  192.88.99.0/24
        Status:
        Events:  <none>
    

    The CIDR ranges are the same as the default non-masquerade destination ranges.

    Managed by GKE policy

    GKE Egress NAT policy reserves a static range of IP addresses required to preserve cluster's operation. This static range contains the Pod, Service, and Node IP address ranges of the cluster, and might overlap with the default policy.

    You can identify this policy by a dynamic 8-byte hash (gke-{CLUSTER_SHORT_HASH}) that GKE assigns. You cannot edit this policy.

    The following manifest describes a managed by GKE policy named gke-bbfa6c0e-1:

        Name:         gke-bbfa6c0e-1
        Namespace:
        Labels:       <none>
        Annotations:  <none>
        API Version:  networking.gke.io/v1
        Kind:         EgressNATPolicy
        Metadata:
          Creation Timestamp:  2022-03-16T21:05:46Z
          Generation:          1
          Managed Fields:
            API Version:  networking.gke.io/v1
            Fields Type:  FieldsV1
            fieldsV1:
              f:spec:
                .:
                f:action:
                f:destinations:
              f:status:
            Manager:         egress-nat-controller
            Operation:       Update
            Time:            2022-03-16T21:05:46Z
          Resource Version:  11699
          UID:               0201b5de-a6f6-4926-822b-31ed7cdee2c6
        Spec:
          Action:  NoSNAT
          Destinations:
            Cidr:  10.119.128.0/17
            Cidr:  10.120.0.0/22
            Cidr:  10.128.0.0/20
        Status:
        Events:  <none>
    

    Masquerading to link-local destinations

    The link-local IP address destinations fit in the 169.254.0.0/16 IP range. By default, GKE preserves source Pod IP addresses for packets sent to 169.254.0.0/16. This default configuration applies to all GKE versions regardless of whether the following conditions are met:

    • The ip-masq-agent DaemonSet is deployed.
    • The nonMasqueradeCIDRs list is specified in the ip-masq-agent ConfigMap.
    • The --disable-default-snat flag is specified when creating the cluster.

    You can configure GKE to change source Pod IP addresses to node IP addresses when sending packets to 169.254.0.0/16 if you deploy the ip-masq-agent DaemonSet and set masqLinkLocal to True in the ip-masq-agent ConfigMap. When masqLinkLocal is not set, GKE assumes masqLinkLocal is False.

    What's next