This page explains how IP masquerading works in GKE and the configuration options for IP masquerade agent.
Overview
IP masquerading is a form of network address translation (NAT) used to perform many-to-one IP address translations, which allows multiple clients to access a destination using a single IP address. A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses.
The following sections describe how to configure IP masquerading in your GKE cluster.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
Set up default gcloud
settings using one of the following methods:
- Using
gcloud init
, if you want to be walked through setting defaults. - Using
gcloud config
, to individually set your project ID, zone, and region.
Using gcloud init
If you receive the error One of [--zone, --region] must be supplied: Please specify
location
, complete this section.
-
Run
gcloud init
and follow the directions:gcloud init
If you are using SSH on a remote server, use the
--console-only
flag to prevent the command from launching a browser:gcloud init --console-only
-
Follow the instructions to authorize
gcloud
to use your Google Cloud account. - Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone.
Using gcloud config
- Set your default project ID:
gcloud config set project PROJECT_ID
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone COMPUTE_ZONE
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region COMPUTE_REGION
- Update
gcloud
to the latest version:gcloud components update
Masquerading in GKE
GKE uses iptables
rules, along with the ip-masq-agent
DaemonSet, to change the source IP address of packets sent from Pods to certain
destinations. When a Pod sends a packet to a destination IP address in a
specified masquerade range, the node's IP address is used as the packet's source
address (instead of the Pod's IP address).
GKE performs IP masquerading as summarized in this table:
Cluster configuration | SNAT behavior |
---|---|
|
GKE preserves the source Pod IP addresses for
packets sent to destinations specified in the
GKE changes source Pod IP addresses to source node IP
addresses for packets sent to destinations not specified in the
|
|
GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. These defaults depend on GKE version and node image type. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations. |
|
GKE preserves the source Pod IP addresses for packets sent to all destinations. Change this behavior by ensuring
|
Additional information is available in the IP Masquerade Agent User Guide in the Kubernetes documentation.
When is ip-masq-agent
included
The ip-masq-agent
DaemonSet is automatically installed as an add-on with
--nomasq-all-reserved-ranges
argument in a GKE cluster, if
one or more of the following is true:
- The cluster has a network policy.
- The Pod's CIDR range is not within
10.0.0.0/8
. - The cluster was created without the
--disable-default-snat
flag, and has Workload Identity enabled.
You can change the destination ranges by specifying the nonMasqueradeCIDRs
in
the ip-masq-agent
ConfigMap. If your cluster doesn't include the
ip-masq-agent
, you can manually install ip-masq-agent
.
Default non-masquerade destinations
The following table summarizes GKE default non-masquerade
destination ranges for clusters created without the
--disable-default-snat
flag, provided one of the following is true:
ip-masq-agent
is not deployed , ornonMasqueradeCIDRs
is not specified in theip-masq-agent
ConfigMap
GKE versions | Node image type | Destination ranges | Notes |
---|---|---|---|
All versions | All image types | 169.254.0.0/16 (link-local range) |
You can change this behavior by setting masqLinkLocal to
True in the ip-masq-agent ConfigMap. |
Versions before 1.14 |
Container-Optimized OS with containerd (cos_containerd)
or ip-masq-agent started with the --nomasq-all-reserved-ranges argument
|
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 |
You can change this behavior by specifying a list of CIDRs in nonMasqueradeCIDRs
in the ip-masq-agent ConfigMap.
See the VPC networks valid ranges for information about the use of non-RFC 1918 reserved address ranges. |
Other image types without ip-masq-agent enabled |
10.0.0.0/8 | ||
Versions 1.14.1-gke.14, 1.14.2-gke.1 and later | All image types |
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4 |
Checking the masquerade rules in iptables
SSH into one of the nodes and run the following command:
iptables -t nat -L IP-MASQ
Here is an example output from a cluster not running the ip-masq-agent
daemonset,
with the node pool on a version older than 1.14 and using a cos_cointainerd image
(see table in the section above for details):
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 172.16.0.0/12 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 192.168.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
ip-masq-agent
ConfigMap parameters
The following keys in the ip-masq-agent
ConfigMap allow you to specify
non-masquerade destination ranges and change the behavior of ip-masq-agent
.
The ConfigMap file must be written in YAML or JSON and must be named config
.
- nonMasqueradeCIDRs: A list of strings in CIDR notation that specify the destination IP address ranges for preserving Pod IP addresses.
- masqLinkLocal: A boolean value which indicates whether
to masquerade traffic to the link-local prefix (
169.254.0.0/16
). Default isfalse
. - resyncInterval: Represents the amount of time that passes before
ip-masq-agent
reloads its configuration (writing to/etc/config/ip-masq-agent
from the ConfigMap). The format isNx
, whereN
is an integer andx
is a time unit such ass
orms
. If unspecified, the default is60
seconds.
Specifying nonMasqueradeCIDRs
The following ConfigMap preserves source address for packets sent to
10.0.0.0/8
:
nonMasqueradeCIDRs:
- 10.0.0.0/8
resyncInterval: 60s
Ignoring the link-local Range
To have ip-masq-agent
enable IP masquerading for the link-local range, set the
value of the masqLinkLocal
key to true
in the ConfigMap.
For example:
nonMasqueradeCIDRs:
- 10.0.0.0/8
resyncInterval: 60s
masqLinkLocal: true
Adding a ConfigMap to your cluster
To add a ConfigMap to your cluster, run the following command from your shell or terminal window:
kubectl create configmap configmap-name \
--from-file config \
--namespace kube-system
where configmap-name is the name you choose for the ConfigMap
resource, such as ip-masq-agent
.
For example:
kubectl create configmap ip-masq-agent --from-file config --namespace kube-system
After the sync is complete, you should see the changes in iptables
:
iptables -t nat -L IP-MASQ
Chain IP-MASQ (1 references)
target prot opt source destination
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local */
MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
Manually creating an ip-masq-agent
resource (optional)
You might need to manually create and configure ip-masq-agent
. To do so, you
need to deploy an ip-masq-agent
resource in your cluster.
Save the following manifest into a file named
ip-masq-agent.yaml
locally:apiVersion: apps/v1 kind: DaemonSet metadata: name: ip-masq-agent namespace: kube-system spec: selector: matchLabels: k8s-app: ip-masq-agent template: metadata: labels: k8s-app: ip-masq-agent spec: hostNetwork: true containers: - name: ip-masq-agent image: gcr.io/google-containers/ip-masq-agent-amd64:latest args: - --masq-chain=IP-MASQ # To non-masquerade reserved IP ranges by default, uncomment the line below. # - --nomasq-all-reserved-ranges securityContext: privileged: true volumeMounts: - name: config mountPath: /etc/config volumes: - name: config configMap: # Note this ConfigMap must be created in the same namespace as the # daemon pods - this spec uses kube-system name: ip-masq-agent optional: true items: # The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent - key: config path: ip-masq-agent tolerations: - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists - key: "CriticalAddonsOnly" operator: "Exists"
Run the following command:
kubectl apply -f ip-masq-agent.yaml
This should create a daemonset named ip-masq-agent
that runs on all nodes in your cluster.
To configure the ip-masq-agent
tool, refer to
Configuring the agent using a ConfigMap.
Restrictions
Clusters running ip-masq-agent
have the following restrictions:
- If you enable intranode visibility,
and use
ip-masq-agent
configured with thenonMasqueradeCIDRs
parameter, thenonMasqueradeCIDRs
needs to include the Pod CIDR, otherwise you can experience intranode connectivity issues.