This page explains how to configure clusters created in Google Kubernetes Engine (GKE)
Standard mode to perform IP masquerade
with the ip-masq-agent
. For more information about IP masquerading in
GKE Autopilot mode,
see Use Egress NAT Policy to configure IP masquerade in Autopilot clusters.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Checking ip-masq-agent
status
This section shows you how to:
- Determine whether your cluster has an
ip-masq-agent
DaemonSet. - Check the
ip-masq-agent
ConfigMap resource.
Checking the ip-masq-agent
DaemonSet
To check if your cluster is running the ip-masq-agent
DaemonSet, use the
Google Cloud CLI or the Google Cloud console.
gcloud
Get the credentials for your cluster:
gcloud container clusters get-credentials CLUSTER_NAME
Replace
CLUSTER_NAME
with the name of your cluster.Search for the
ip-masq-agent
in thekube-system
namespace:kubectl get daemonsets/ip-masq-agent -n kube-system
If the
ip-masq-agent
DaemonSet exists, then the output is similar to the following:NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE ip-masq-agent 3 3 3 3 3 <none> 13d
If the
ip-masq-agent
DaemonSet does not exist, then the output is similar to the following:Error from server (NotFound): daemonsets.apps "ip-masq-agent" not found
Console
Go to the Workloads page in the Google Cloud console.
For
Filter, do the following:- Click to clear the Is system object: False filter.
- Filter the following properties:
- Name:
ip-masq-agent
. - Cluster: the name of your cluster.
- Name:
If
ip-masq-agent
DaemonSet exists, then you can see the DaemonSet record in the table. Ifip-masq-agent
DaemonSet does not exist, then no rows are displayed.
To create the ip-masq-agent
ConfigMap and deploy the ip-masq-agent
DaemonSet,
refer to Configuring and deploying the ip-masq-agent
.
Checking the ip-masq-agent
ConfigMap
To check if your cluster is running the ip-masq-agent
ConfigMap, use the
Google Cloud CLI or the Google Cloud console.
gcloud
Get the credentials for your cluster:
gcloud container clusters get-credentials CLUSTER_NAME
Replace
CLUSTER_NAME
with the name of your cluster.Describe the
ip-masq-agent
ConfigMap in thekube-system
namespace:kubectl describe configmaps/ip-masq-agent -n kube-system
If the
ip-masq-agent
ConfigMap exists, then the output is similar to the following:Name: ip-masq-agent Namespace: kube-system Labels: <none> Annotations: <none> Data ==== config: ---- nonMasqueradeCIDRs: - 198.15.5.92/24 - 10.0.0.0/8 masqLinkLocal: false resyncInterval: 60s BinaryData ==== Events: <none>
If the
ip-masq-agent
ConfigMap does not exist, then the output is similar to the following:Error from server (NotFound): configmaps "ip-masq-agent" not found
Console
Go to the Configuration page in the Google Cloud console.
For
Filter, do the following:- Click to clear the Is system object: False filter.
- Filter the following properties:
- Name:
ip-masq-agent
. - Cluster: the name of your cluster.
- Name:
If
ip-masq-agent
ConfigMap exists, then you can see the ConfigMap record in the table. Ifip-masq-agent
ConfigMap does not exist, then no rows are displayed.
If the cluster already has the ip-masq-agent
ConfigMap, you can configure and
deploy it.
Configuring and deploying the ip-masq-agent
This section shows you how to create or edit the ip-masq-agent
ConfigMap and how to deploy or delete the ip-masq-agent
DaemonSet. To
determine which tasks you need to perform, you must first determine whether your
cluster already has the ip-masq-agent
ConfigMap and ip-masq-agent
DaemonSet.
Creating the ip-masq-agent
ConfigMap
The following steps show how to create the ip-masq-agent
ConfigMap. If
your cluster already has the ip-masq-agent
ConfigMap, edit an existing
ip-masq-agent ConfigMap instead.
Create a configuration file using the following template and save it locally. You can use any name for the local copy of this configuration file.
nonMasqueradeCIDRs: - CIDR_1 - CIDR_2 masqLinkLocal: false resyncInterval: SYNC_INTERVALUNIT_OF_TIME
Replace the following:
CIDR_1
andCIDR_2
: the IP address ranges in CIDR format. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to thenonMasqueradeCIDRs
list following the same format. At minimum, thenonMasqueradeCIDRs
property should include the node and Pod IP address ranges of your cluster.SYNC_INTERVAL
: the amount of time after which eachip-masq-agent
Pod checks the contents of theip-masq-agent
ConfigMap and writes any changes to its local/etc/config/ip-masq-agent
file. Defaults to60
.UNIT_OF_TIME
: the unit of time for the resyncInterval. Valid values includes
(for seconds) orms
(for milliseconds). Defaults tos
.
Set
masqLinkLocal
to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations.Create the ConfigMap resource:
kubectl create configmap ip-masq-agent \ --namespace=kube-system \ --from-file=config=LOCAL_CONFIG_FILE_PATH
Replace
LOCAL_CONFIG_FILE_PATH
with the path to the config file you created in the previous step.Describe the
ip-masq-agent
ConfigMap in thekube-system
namespace:kubectl describe configmaps/ip-masq-agent -n kube-system
The output is similar to the following:
Name: ip-masq-agent Namespace: kube-system Labels: <none> Annotations: <none> Data ==== config: ---- nonMasqueradeCIDRs: - 198.15.5.92/24 - 10.0.0.0/8 masqLinkLocal: false resyncInterval: 60s BinaryData ==== Events: <none>
This output includes the
config
parameter with your configuration changes. You can now deploy theip-masq-agent
DeamonSet.
Editing an existing ip-masq-agent
ConfigMap
You can modify the contents of an existing ip-masq-agent
ConfigMap by
completing the following steps:
Open the ConfigMap in a text editor:
kubectl edit configmap ip-masq-agent --namespace=kube-system
Edit the content of the ConfigMap file:
apiVersion: v1 data: config: | nonMasqueradeCIDRs: - CIDR_1 - CIDR_2 masqLinkLocal: false resyncInterval: SYNC_INTERVALUNIT_OF_TIME kind: ConfigMap metadata: name: ip-masq-agent namespace: kube-system
Replace the following:
CIDR_1
andCIDR_2
: the IP address ranges in CIDR format. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to thenonMasqueradeCIDRs
list following the same format. At minimum, thenonMasqueradeCIDRs
property should include the node and Pod IP address ranges of your cluster.SYNC_INTERVAL
: the amount of time after which eachip-masq-agent
Pod checks the contents of theip-masq-agent
ConfigMap and writes any changes to its local/etc/config/ip-masq-agent
file. Defaults to60
.UNIT_OF_TIME
: the unit of time for the resyncInterval. Valid values includes
(for seconds) orms
(for milliseconds). Defaults tos
.
Set
masqLinkLocal
to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations.Describe the
ip-masq-agent
ConfigMap in thekube-system
namespace:kubectl describe configmaps/ip-masq-agent -n kube-system
The output is similar to the following:
Name: ip-masq-agent Namespace: kube-system Labels: <none> Annotations: <none> Data ==== config: ---- nonMasqueradeCIDRs: - 198.15.5.92/24 - 10.0.0.0/8 masqLinkLocal: false resyncInterval: 60s BinaryData ==== Events: <none>
This output includes the
config
parameter which matches the configuration value from the file you created.
Deploying the ip-masq-agent
DaemonSet
After you create or edit your ip-masq-agent
ConfigMap, deploy the
ip-masq-agent
DaemonSet.
Save the following manifest as a YAML file:
apiVersion: apps/v1 kind: DaemonSet metadata: name: ip-masq-agent namespace: kube-system spec: selector: matchLabels: k8s-app: ip-masq-agent template: metadata: labels: k8s-app: ip-masq-agent spec: hostNetwork: true containers: - name: ip-masq-agent image: gke.gcr.io/ip-masq-agent:v2.11.0-gke.23@sha256:f50757332ee45c0fb9f5856552a3ed16548c1de3c33c4e520a02d22e30af96ae args: # The masq-chain must be IP-MASQ - --masq-chain=IP-MASQ # To non-masquerade reserved IP ranges by default, # uncomment the following line. # - --nomasq-all-reserved-ranges # Must be set to false when using Dataplane V2. - --random-fully=false securityContext: privileged: false capabilities: drop: ["ALL"] add: ["NET_ADMIN", "NET_RAW"] allowPrivilegeEscalation: false seccompProfile: type: RuntimeDefault volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: ip-masq-agent optional: true items: - key: config path: ip-masq-agent tolerations: - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists - key: "CriticalAddonsOnly" operator: "Exists"
This manifest creates a volume named
config-volume
which is mounted as specified by the container's volumeMount.If you need to edit this manifest, consider the following conditions:
The volume name can be anything but must match the container's
volumeMount
name.The ConfigMap name must match the name of the
configMap
referenced in theconfig-volume
Volume in the Pod.The name of the chain (
--masq-chain
) must beIP-MASQ
. Otherwise, GKE does not override the default masquerading rules.DaemonSet Pods read from the
ip-masq-agent
file. Theip-masq-agent
file content is the value of theconfig
key in the ConfigMap.If you use non-masquerade reserved IP ranges by default, uncomment the
- --nomasq-all-reserved-ranges
line in thearg
section.
Deploy the DaemonSet:
kubectl apply -f LOCAL_FILE_PATH
Replace
LOCAL_FILE_PATH
with the path to the file you created in the previous step.
You can manually update the ip-masq-agent
DaemonSet you created. To learn more, see
Updating a DaemonSet.
Deleting the ip-masq-agent
This section shows you how to delete the ip-masq-agent
DaemonSet and the
ip-masq-agent
ConfigMap. Deleting the ip-masq-agent
doesn't revert the existing
IP masquerading settings on the nodes.
Deleting the ip-masq-agent
DaemonSet
If you manually created the ip-masq-agent
DaemonSet, you can delete it by
running the following command:
kubectl delete daemonsets ip-masq-agent -n kube-system
Deleting the ip-masq-agent
ConfigMap
To completely delete the ip-masq-agent
ConfigMap, run the following command:
kubectl delete configmap ip-masq-agent -n kube-system
Troubleshooting
The following steps provide troubleshooting information:
- Confirm the status of the
ip-masq-agent
. If the ConfigMap is not defined, traffic to all default destinations is not masqueraded and preserves the Pod IP address. Traffic to other destinations preserves the node IP address. - Check whether the IP-MASQ chain is correctly populated in the NAT IP tables by running the
sudo iptables -t nat -L IP-MASQ
command in the impacted node. If thenonMasqueradeCIDRs
defined in the ConfigMap is not showing up on the NAT IP Tables, verify that there are no typos in the config file that was used to create the ConfigMap. - Confirm that the destination the Pod is trying to reach is included in the
nonMasqueradeCIDRs
in the ConfigMap. If the destination is not in thenonMasqueradeCIDRs
, traffic preserves the node IP address. - Confirm that the destination allows both the node and Pod IP address ranges.
- If traffic is not accessible from the node or Pod, run a connectivity test.