Using an IP masquerade agent

This page explains how IP masquerading works in GKE and the configuration options for IP masquerade agent.

Overview

IP masquerading is a form of network address translation (NAT) used to perform many-to-one IP address translations, which allows multiple clients to access a destination using a single IP address. A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses.

The following sections describe how to configure IP masquerading in your GKE cluster.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project PROJECT_ID

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone COMPUTE_ZONE

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region COMPUTE_REGION

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Masquerading in GKE

GKE uses iptables rules, along with the ip-masq-agent DaemonSet, to change the source IP address of packets sent from Pods to certain destinations. When a Pod sends a packet to a destination IP address in a specified masquerade range, the node's IP address is used as the packet's source address (instead of the Pod's IP address).

GKE performs IP masquerading as summarized in this table:

Cluster configuration	SNAT behavior
The ip-masq-agent is installed in your cluster, and You have specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap	GKE preserves the source Pod IP addresses for packets sent to destinations specified in the nonMasqueradeCIDRs list. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations not specified in the nonMasqueradeCIDRs list.
Either ip-masq-agent is not installed or you have not specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap, and You have created the cluster without the --disable-default-snat flag	GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. These defaults depend on GKE version and node image type. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations.
Either ip-masq-agent is not installed or you have not specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap, and You have created the cluster with the --disable-default-snat flag	GKE preserves the source Pod IP addresses for packets sent to all destinations. Change this behavior by ensuring ip-masq-agent is installed and you have specified a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap.

Additional information is available in the IP Masquerade Agent User Guide in the Kubernetes documentation.

When is ip-masq-agent included

The ip-masq-agent DaemonSet is automatically installed as an add-on with --nomasq-all-reserved-ranges argument in a GKE cluster, if one or more of the following is true:

• The cluster has a network policy.
• The Pod's CIDR range is not within 10.0.0.0/8.
• The cluster was created without the --disable-default-snat flag, and has Workload Identity enabled.

You can change the destination ranges by specifying the nonMasqueradeCIDRs in the ip-masq-agent ConfigMap. If your cluster doesn't include the ip-masq-agent, you can manually install ip-masq-agent.

Default non-masquerade destinations

The following table summarizes GKE default non-masquerade destination ranges for clusters created without the --disable-default-snat flag, provided one of the following is true:

• ip-masq-agent is not deployed , or
• nonMasqueradeCIDRs is not specified in the ip-masq-agent ConfigMap

GKE versions	Node image type	Destination ranges	Notes
All versions	All image types	169.254.0.0/16 (link-local range)	You can change this behavior by setting masqLinkLocal to True in the ip-masq-agent ConfigMap.
Versions before 1.14	Container-Optimized OS with containerd (cos_containerd) or ip-masq-agent started with the --nomasq-all-reserved-ranges argument	10.0.0.0/8 172.16.0.0/12 192.168.0.0/16	You can change this behavior by specifying a list of CIDRs in nonMasqueradeCIDRs in the ip-masq-agent ConfigMap. See the VPC networks valid ranges for information about the use of non-RFC 1918 reserved address ranges.
	Other image types without ip-masq-agent enabled	10.0.0.0/8
Versions 1.14.1-gke.14, 1.14.2-gke.1 and later	All image types	10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4

Note: If the ip-masq-agent daemonset is running, either as an add-on or installed manually, the masquerading behavior described in the table above depends on the control plane version of the cluster. If the ip-masq-agent is not running, the source of the masq rules is then the node startup script. Hence, the masquerading behavior then depends on the node pool version. A GKE cluster with different node pool versions could have different masq rules for the respective node pools.

Checking the masquerade rules in iptables

SSH into one of the nodes and run the following command:

iptables -t nat -L IP-MASQ

Here is an example output from a cluster not running the ip-masq-agent daemonset, with the node pool on a version older than 1.14 and using a cos_cointainerd image (see table in the section above for details):

RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             172.16.0.0/12        /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             192.168.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE  all  --  anywhere            anywhere             /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

ip-masq-agent ConfigMap parameters

The following keys in the ip-masq-agent ConfigMap allow you to specify non-masquerade destination ranges and change the behavior of ip-masq-agent. The ConfigMap file must be written in YAML or JSON and must be named config.

• nonMasqueradeCIDRs: A list of strings in CIDR notation that specify the destination IP address ranges for preserving Pod IP addresses.
• masqLinkLocal: A boolean value which indicates whether to masquerade traffic to the link-local prefix (169.254.0.0/16). Default is false.
• resyncInterval: Represents the amount of time that passes before ip-masq-agent reloads its configuration (writing to /etc/config/ip-masq-agent from the ConfigMap). The format is Nx, where N is an integer and x is a time unit such as s or ms. If unspecified, the default is 60 seconds.

Specifying nonMasqueradeCIDRs

The following ConfigMap preserves source address for packets sent to 10.0.0.0/8:

nonMasqueradeCIDRs:
  - 10.0.0.0/8
resyncInterval: 60s

Ignoring the link-local Range

To have ip-masq-agent enable IP masquerading for the link-local range, set the value of the masqLinkLocal key to true in the ConfigMap.

For example:

nonMasqueradeCIDRs:
  - 10.0.0.0/8
resyncInterval: 60s
masqLinkLocal: true

Adding a ConfigMap to your cluster

Note: The ConfigMap file you create for ip-masq-agent must be named config.

To add a ConfigMap to your cluster, run the following command from your shell or terminal window:

kubectl create configmap configmap-name \
  --from-file config \
  --namespace kube-system

where configmap-name is the name you choose for the ConfigMap resource, such as ip-masq-agent.

For example:

kubectl create configmap ip-masq-agent --from-file config --namespace kube-system

After the sync is complete, you should see the changes in iptables:

iptables -t nat -L IP-MASQ

Chain IP-MASQ (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local */
MASQUERADE  all  --  anywhere            anywhere             /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

Manually creating an ip-masq-agent resource (optional)

You might need to manually create and configure ip-masq-agent. To do so, you need to deploy an ip-masq-agent resource in your cluster.

1. Save the following manifest into a file named ip-masq-agent.yaml locally:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: ip-masq-agent
     namespace: kube-system
   spec:
     selector:
       matchLabels:
         k8s-app: ip-masq-agent
     template:
       metadata:
         labels:
           k8s-app: ip-masq-agent
       spec:
         hostNetwork: true
         containers:
         - name: ip-masq-agent
           image: gcr.io/google-containers/ip-masq-agent-amd64:latest
           args:
               - --masq-chain=IP-MASQ
               # To non-masquerade reserved IP ranges by default, uncomment the line below.
               # - --nomasq-all-reserved-ranges
           securityContext:
             privileged: true
           volumeMounts:
             - name: config
               mountPath: /etc/config
         volumes:
           - name: config
             configMap:
               # Note this ConfigMap must be created in the same namespace as the
               # daemon pods - this spec uses kube-system
               name: ip-masq-agent
               optional: true
               items:
                 # The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent
                 - key: config
                   path: ip-masq-agent
         tolerations:
         - effect: NoSchedule
           operator: Exists
         - effect: NoExecute
           operator: Exists
         - key: "CriticalAddonsOnly"
           operator: "Exists"
   

2. Run the following command:

   kubectl apply -f ip-masq-agent.yaml
   

This should create a daemonset named ip-masq-agent that runs on all nodes in your cluster.

To configure the ip-masq-agent tool, refer to Configuring the agent using a ConfigMap.

Restrictions

Clusters running ip-masq-agent have the following restrictions:

• If you enable intranode visibility, and use ip-masq-agent configured with the nonMasqueradeCIDRs parameter, the nonMasqueradeCIDRs needs to include the Pod CIDR, otherwise you can experience intranode connectivity issues.

What's next

• Learn about Alias IPs.
• Read the GKE network overview.
• Learn about configuring authorized networks.