IP masquerade agent

This page explains how IP masquerading works in GKE and the configuration options for IP masquerade agent.

Overview

IP masquerading is a form of network address translation (NAT) used to perform many-to-one IP address translations, which allows multiple clients to access a destination using a single IP address. A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses.

The following sections describe how to configure IP masquerading in your GKE cluster.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone.

Using gcloud config

  • Set your default project ID:
    gcloud config set project project-id
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone compute-zone
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region compute-region
  • Update gcloud to the latest version:
    gcloud components update

Masquerading in GKE

GKE uses iptables rules, along with the ip-masq-agent DaemonSet, to change the source IP address of packets sent from Pods to certain destinations. When a Pod sends a packet to a destination IP address in a specified masquerade range, the node's IP address is used as the packet's source address (instead of the Pod's IP address).

GKE performs IP masquerading as summarized in this table:

Cluster configuration SNAT behavior

GKE preserves the source Pod IP addresses for packets sent to destinations specified in the nonMasqueradeCIDRs list.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in the nonMasqueradeCIDRs list.

  • Either ip-masq-agent is not installed or you have not specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap, and
  • You have created the cluster without the
    --disable-default-snat flag

GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. These defaults depend on GKE version and node image type.

GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.

  • Either ip-masq-agent is not installed or you have not specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap, and
  • You have created the cluster with the
    --disable-default-snat flag

GKE preserves the source Pod IP addresses for packets sent to all destinations.

Change this behavior by ensuring ip-masq-agent is installed and you have specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap.

Additional information is available in the IP Masquerade Agent User Guide in the Kubernetes documentation.

When is ip-masq-agent included

The ip-masq-agent DaemonSet is automatically installed as an add-on with --nomasq-all-reserved-ranges argument in a GKE cluster, if one or more of the following is true:

  • The cluster has a network policy.
  • The Pod's CIDR range is not within 10.0.0.0/8.

You can change the destination ranges by specifying the nonMasqueradeCIDRs in the ip-masq-agent ConfigMap. If your cluster doesn't include the ip-masq-agent, you can manually install ip-masq-agent.

Default non-masquerade destinations

The following table summarizes GKE default non-masquerade destination ranges for clusters created without the --disable-default-snat flag, provided one of the following is true:

  • ip-masq-agent is not deployed , or
  • nonMasqueradeCIDRs is not specified in the ip-masq-agent ConfigMap
GKE versions Node image type Destination ranges Notes
All versions All image types 169.254.0.0/16
(link-local range)
You can change this behavior by setting masqLinkLocal to True in the ip-masq-agent ConfigMap.
Versions before 1.14 Container-Optimized OS with containerd (cos_containerd)
or ip-masq-agent started with the --nomasq-all-reserved-ranges argument
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
You can change this behavior by specifying a list of CIDRs in nonMasqueradeCIDRs in the ip-masq-agent ConfigMap.

See the VPC networks valid ranges for information about the use of non-RFC 1918 reserved address ranges.
Other image types without ip-masq-agent enabled 10.0.0.0/8
Versions 1.14.1-gke.14, 1.14.2-gke.1 and later All image types 10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
100.64.0.0/10
192.0.0.0/24
192.0.2.0/24
192.88.99.0/24
198.18.0.0/15
198.51.100.0/24
203.0.113.0/24
240.0.0.0/4

Checking the masquerade rules in iptables

SSH into one of the nodes and run the following command:

iptables -t nat -L IP-MASQ

Here is an example output from a cluster not running the ip-masq-agent daemonset, with the node pool on a version older than 1.14 and using a cos_cointainerd image (see table in the section above for details):

RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             172.16.0.0/12        /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             192.168.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE  all  --  anywhere            anywhere             /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

ip-masq-agent ConfigMap parameters

The following keys in the ip-masq-agent ConfigMap allow you to specify non-masquerade destination ranges and change the behavior of ip-masq-agent. The ConfigMap file must be written in YAML or JSON and must be named config.

  • nonMasqueradeCIDRs: A list of strings in CIDR notation that specify the destination IP address ranges for preserving Pod IP addresses.
  • masqLinkLocal: A boolean value which indicates whether to masquerade traffic to the link-local prefix (169.254.0.0/16). Default is false.
  • resyncInterval: Represents the amount of time that passes before ip-masq-agent reloads its configuration (writing to /etc/config/ip-masq-agent from the ConfigMap). The format is Nx, where N is an integer and x is a time unit such as s or ms. If unspecified, the default is 60 seconds.

Specifying nonMasqueradeCIDRs

The following ConfigMap, config, preserves source address for packets sent to 10.0.0.0/8:

nonMasqueradeCIDRs:
  - 10.0.0.0/8
resyncInterval: 60s

To have ip-masq-agent enable IP masquerading for the link-local range, set the value of the masqLinkLocal key to true in the ConfigMap.

For example:

nonMasqueradeCIDRs:
  - 10.0.0.0/8
resyncInterval: 60s
masqLinkLocal: true

Adding a ConfigMap to your cluster

To add a ConfigMap to your cluster, run the following command from your shell or terminal window:

kubectl create configmap configmap-name \
  --from-file config \
  --namespace kube-system

where configmap-name is the name you choose for the ConfigMap resource, such as ip-masq-agent.

For example:

kubectl create configmap ip-masq-agent --from-file config --namespace kube-system

After the sync is complete, you should see the changes in iptables:

iptables -t nat -L IP-MASQ
Chain IP-MASQ (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local */
MASQUERADE  all  --  anywhere            anywhere             /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

Manually creating an ip-masq-agent resource (optional)

You might need to manually create and configure ip-masq-agent. To do so, you need to deploy an ip-masq-agent resource in your cluster.

  1. Save the following manifest into a file named ip-masq-agent.yaml locally:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: ip-masq-agent
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          k8s-app: ip-masq-agent
      template:
        metadata:
          labels:
            k8s-app: ip-masq-agent
        spec:
          hostNetwork: true
          containers:
          - name: ip-masq-agent
            image: gcr.io/google-containers/ip-masq-agent-amd64:v2.4.1
            args:
                - --masq-chain=IP-MASQ
                # To non-masquerade reserved IP ranges by default, uncomment the line below.
                # - --nomasq-all-reserved-ranges
            securityContext:
              privileged: true
            volumeMounts:
              - name: config
                mountPath: /etc/config
          volumes:
            - name: config
              configMap:
                # Note this ConfigMap must be created in the same namespace as the
                # daemon pods - this spec uses kube-system
                name: ip-masq-agent
                optional: true
                items:
                  # The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent
                  - key: config
                    path: ip-masq-agent
          tolerations:
          - effect: NoSchedule
            operator: Exists
          - effect: NoExecute
            operator: Exists
          - key: "CriticalAddonsOnly"
            operator: "Exists"
    
  2. Run the following command:

    kubectl apply -f ip-masq-agent.yaml
    

This should create a daemonset named ip-masq-agent that runs on all nodes in your cluster.

To configure the ip-masq-agent tool, refer to Configuring the agent using a ConfigMap.

Restrictions

Clusters running ip-masq-agent have the following restrictions:

  • If you enable intranode visibility, and use ip-masq-agent configured with the nonMasqueradeCIDRs parameter, the nonMasqueradeCIDRs needs to include the Pod CIDR, otherwise you can experience intranode connectivity issues.

What's next