This page explains how IP masquerading works in GKE and the configuration options for IP masquerade agent.
Overview
IP masquerading is a form of network address translation (NAT) used to perform many-to-one IP address translations, which allows multiple clients to access a destination using a single IP address. A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses.
The following sections describe how to configure IP masquerading in your GKE cluster.
Before you begin
To prepare for this task, perform the following steps:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
- Set your default project ID:
gcloud config set project [PROJECT_ID]
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone [COMPUTE_ZONE]
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region [COMPUTE_REGION]
- Update
gcloud
to the latest version:gcloud components update
Masquerading in GKE
GKE uses iptables
rules, along with the ip-masq-agent
DaemonSet, to change the source IP address of packets sent from Pods to certain
destinations. When a Pod sends a packet to a destination IP address in a
specified masquerade range, the node's IP address is used as the packet's source
address (instead of the Pod's IP address).
GKE performs IP masquerading for all destinations except:
If
ip-masq-agent
is installed in your cluster and you defined your ownnonMasqueradeCIDRs
in theip-masq-agent
ConfigMap, then GKE preserves the source IP addresses as Pod IP addresses for packets sent to destinations specified innonMasqueradeCIDRs
.If
ip-masq-agent
is not installed or if no CIDRs are specified innonMasqueradeCIDRs
, then GKE preserves the source IP addresses for packets sent to a set of default non-masquerade destinations. These defaults depend on GKE version and node image type.
Additional information is available in the IP Masquerade Agent User Guide in the Kubernetes documentation.
When ip-masq-agent
is included
The ip-masq-agent
DaemonSet is automatically installed with
--nomasq-all-reserved-ranges
argument in GKE cluster, if
one or more of the following is true:
- The cluster has a network policy
- The Pod's CIDR range is not within
10.0.0.0/8
You can change the destination ranges by specifying the nonMasqueradeCIDRs
in
the ip-agent-masq
ConfigMap. If your cluster doesn't include the
ip-masq-agent
, you can manually install ip-masq-agent
.
Default non-masquerade destinations
The following table summarizes GKE default non-masquerade
destination ranges when nonMasqueradeCIDRs
is not specified in the
ip-masq-agent
ConfigMap.
GKE versions | Node image type | Destination ranges | Notes |
---|---|---|---|
All versions | All image types | 169.254.0.0/16 (link-local range) |
You can change this behavior by setting masqLinkLocal to
True in the ip-masq-agent ConfigMap. |
Versions before 1.14 |
Container-Optimized OS with containerd (cos_containerd)
or ip-masq-agent started with the --nomasq-all-reserved-ranges argument
|
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 |
You can change this behavior by specifying a list of CIDRs in nonMasqueradeCIDRs
in the ip-masq-agent ConfigMap. |
Other image types without ip-masq-agent enabled |
10.0.0.0/8 | ||
Versions 1.14.1-gke.14, 1.14.2-gke.1 and later | All image types |
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4 |
Example iptables
output
The follow example demonstrates iptables
rules in the NAT
table, as
maintained by ip-masq-agent
:
iptables -t nat -L IP-MASQ
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL RETURN all -- anywhere 172.16.0.0/12 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL RETURN all -- anywhere 192.168.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
ip-masq-agent
ConfigMap parameters
The following keys in the ip-masq-agent
ConfigMap allow you to specify
non-masquerade destination ranges and change the behavior of ip-masq-agent
.
The ConfigMap file must be written in YAML or JSON and must be named config
.
nonMasqueradeCIDRs
: A list of strings in CIDR notation that specify the destination IP address ranges for preserving Pod IP addresses.masqLinkLocal
: A boolean value which indicates whether to masquerade traffic to the link-local prefix (169.254.0.0/16
). Default isfalse
.resyncInterval
: Represents the amount of time that passes beforeip-masq-agent
reloads its configuration (writing to/etc/config/ip-masq-agent
from the ConfigMap). The format isNx
, whereN
is an integer andx
is a time unit such ass
orms
. If unspecified, the default is60
seconds.
Specifying nonMasqueradeCIDRs
The following ConfigMap, config
, preserves source address for packets sent to
10.0.0.0/8
:
nonMasqueradeCIDRs: - 10.0.0.0/8 resyncInterval: 60s
Ignoring the link-local Range
To have ip-masq-agent
enable IP masquerading for the link-local range, set the
value of the masqLinkLocal
key to
true
in the ConfigMap.
For example:
nonMasqueradeCIDRs: - 10.0.0.0/8 resyncInterval: 60s masqLinkLocal: true
Adding a ConfigMap to your Cluster
To add a ConfigMap to your cluster, run the following command from your shell or terminal window:
kubectl create configmap [CONFIGMAP_NAME] \ --from-file config \ --namespace kube-system
where:
[CONFIGMAP_NAME]
is you choose for the ConfigMap resource, such asip-masq-agent
.
For example:
kubectl create configmap ip-masq-agent --from-file config --namespace kube-system
After the sync is complete, you should see the changes in iptables
:
iptables -t nat -L IP-MASQ
Chain IP-MASQ (1 references)
target prot opt source destination
RETURN all -- anywhere 169.254.0.0/16 /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN all -- anywhere 10.0.0.0/8 /* ip-masq-agent: cluster-local */
MASQUERADE all -- anywhere anywhere /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
Manually Creating an ip-masq-agent
Resource (optional)
You might need to manually create and configure ip-masq-agent
. To do so, you
need to deploy a ip-masq-agent
resource in your cluster.
First, save the following into a file named ip-masq-agent.yaml
locally.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: ip-masq-agent namespace: kube-system spec: template: metadata: labels: k8s-app: ip-masq-agent spec: hostNetwork: true containers: - name: ip-masq-agent image: gcr.io/google-containers/ip-masq-agent-amd64:v2.4.1 args: - --masq-chain=IP-MASQ # To non-masquerade reserved IP ranges by default, uncomment the line below. # - --nomasq-all-reserved-ranges securityContext: privileged: true volumeMounts: - name: config mountPath: /etc/config volumes: - name: config configMap: # Note this ConfigMap must be created in the same namespace as the daemon pods - this spec uses kube-system name: ip-masq-agent optional: true items: # The daemon looks for its config in a YAML file at /etc/config/ip-masq-agent - key: config path: ip-masq-agent tolerations: - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists - key: "CriticalAddonsOnly" operator: "Exists"
Then, run the following command:
kubectl apply -f ip-masq-agent.yaml
This should create a daemonset named ip-masq-agent
that runs on all nodes in your cluster.
To configure the ip-masq-agent
tool, refer to
Configuring the agent using a ConfigMap.