This page explains how IP masquerading works in Google Kubernetes Engine (GKE) and provides configuration options for different scenarios.
Overview of IP masquerading
IP masquerading is a form of source network address translation (SNAT) used to perform many-to-one IP address translations. GKE can use IP masquerading to change the source IP addresses of packets sent from Pods. When IP masquerading applies to a packet emitted by a Pod, GKE changes the packet's source address from the Pod IP to the underlying node's IP address. Masquerading a packet's source is useful when a recipient is configured to receive packets only from the cluster's node IP addresses.
On Linux nodes, GKE configures iptables
rules. GKE
uses the ip-masq-agent
DaemonSet
to configure the appropriate dataplane. IP masquerading is not supported with
Windows Server node pools.
For a more general overview of IP masquerading in any Kubernetes implementation, see IP Masquerade Agent User Guide.
IP masquerading in GKE
GKE performs IP masquerading or SNAT under the following conditions:
For clusters in Autopilot mode:
- GKE version is 1.23.4-gke.1600 or later, or 1.22.7-gke.1500 or later.
- The cluster was created with GKE Dataplane V2 enabled.
- The Egress NAT policy was deployed.
For clusters in Standard mode:
- The
ip-masq-agent
DaemonSet is installed. - The
ip-masq-agent
ConfigMap includesnonMasqueradeCIDRs
andmasqLinkLocal
parameters.
The following table describes valid masquerading outcomes based on the cluster mode and configuration:
Cluster mode | Cluster configuration | SNAT behavior |
---|---|---|
Autopilot |
|
GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. GKE preserves the source Pod IP addresses for packets sent to the internal IP ranges for Pods, Services, and nodes on the cluster. You can identify these destinations with the For more information, see Egress NAT policy. |
Standard |
|
GKE preserves the source Pod IP addresses for
packets sent to destinations specified in the
GKE changes source Pod IP addresses to source node IP
addresses for packets sent to destinations not specified in the
|
Standard |
and |
GKE preserves the source Pod IP addresses for packets sent to a set of default non-masquerade destinations. GKE changes source Pod IP addresses to source node IP addresses for packets sent to destinations outside of the default non-masquerade destinations. |
Standard |
and |
GKE preserves the source Pod IP addresses for packets sent to all destinations. Change this behavior by ensuring
|
Masquerading scenarios
The following table summarizes possible SNAT scenarios based on cluster mode, expected behavior, and corresponding GKE cluster configuration:
Cluster mode | Scenario | SNAT behavior | Cluster configuration |
---|---|---|---|
Autopilot | Not masquerade to any destinations in Autopilot mode clusters |
The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations without affecting the cluster communication between nodes, pods, and services. |
0.0.0.0/0 range.
For more information, see how to deploy a new Egress NAT policy. |
Autopilot | Not masquerade outbound traffic to on-premises with PUPI (privately used public IP) ranges. |
Pod IP is accessible from on-premises networks and has routing and firewall rules set up to allow it. You can preserve the source IP address for traffic coming from pods. |
Create a new Egress NAT policy that contains the privately used public IP (PUPI) addresses. For more information, see how to deploy a new Egress NAT policy. |
Standard |
Masquerade to destinations out of 100.64.0.0/24 and 10.0.0.0/8
|
100.64.0.0/24 and 10.0.0.0/8 . |
ip-masq-agent DaemonSet.
nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap to
[100.64.0.0/24,10.0.0.0/8] .
For more information, see how to create the |
Standard | Masquerade to destinations outside of the default non-masquerade destinations. |
The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to the default non-masquerade destinations. |
--disable-default-snat flag.
For more information, see how to delete the |
Standard | Not masquerade to any destinations in Standard mode clusters |
The cluster preserves source Pod IP addresses (non-masquerading) for packets sent to all destinations. |
Configure your cluster as one of the following options:
For more information, see how to configure the |
Diagnostic containers and Pods with hostNetwork: true
IP masquerading is not applicable to packets sent from the following containers. Unless you specify a custom source IP address for packets, these containers send packets with sources set to the node's primary internal IP address:
- Diagnostic containers for debugging node issues using toolbox.
- Containers in Pods with
spec.hostNetwork: true
When is the ip-masq-agent
installed
The ip-masq-agent
DaemonSet is automatically installed on nodes of your
cluster if you created the cluster without the --disable-default-snat
flag,
and one or more of the following conditions are true:
- The cluster has network policy enabled.
- The cluster's Pod IP address range does not match or is not within
10.0.0.0/8
.
For the ip-masq-agent
DaemonSet to be effective, you must also
specify the nonMasqueradeCIDRs
list
in the ip-masq-agent
ConfigMap. For more information, see
how to configure an IP masquerade agent.
GKE constantly updates and reconciles the ip-masq-agent
DaemonSet.
Default non-masquerade destinations
The following table summarizes GKE default non-masquerade destination ranges.
GKE versions | Destination ranges |
---|---|
Versions earlier than 1.14 | Varies by version and image type. |
Versions equal to or later than:
|
10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 192.0.0.0/24 192.0.2.0/24 192.88.99.0/24 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4
|
The default non-masquerade destinations are only relevant when the cluster meets the following conditions:
- You created the cluster without the
--disable-default-snat
flag , and - One or both of the following are true:
- You have not deployed the
ip-masq-agent
DaemonSet. - The
nonMasqueradeCIDRs
list is not specified in theip-masq-agent
ConfigMap. This also includes the case where you don't specify theip-masq-agent
ConfigMap file.
- You have not deployed the
For a cluster meeting the preceding conditions, GKE takes the following actions:
- The cluster preserves source Pod IP addresses if the packet's destination is in the list of default non-masquerade destinations.
- The cluster changes the source IP addresses from the Pod IP address to the IP address of the node running the Pod, if the packet's destination is not in the list of default non-masquerade destinations.
If you configure the cluster where both the ip-masq-agent
DaemonSet is
deployed and a nonMasqueradeCIDRs
list is specified in the ip-masq-agent
ConfigMap, then the default non-masquerade destinations no longer apply. This is
because the non-masquerade destinations are specified in the
nonMasqueradeCIDRs
list instead.
Effect of the --disable-default-snat
flag
The following describe the purpose of the --disable-default-snat
flag:
When you create a cluster with the
--disable-default-snat
flag, GKE preserves source Pod IP addresses for packets sent to all destinations if one or both of the following are true:- The
ip-masq-agent
DaemonSet is not deployed. - The
nonMasqueradeCIDRs
list is not specified in theip-masq-agent
ConfigMap (including the case where theip-masq-agent
ConfigMap is not specified at all).
If your cluster meets either or both of these conditions, then GKE preserves source Pod IP addresses for packets sent to any destination.
- The
If your cluster has Pods which use privately used public IP addresses, you must create the cluster with the
--disable-default-snat
flag.The
--disable-default-snat
flag does not override a validip-masq-agent
DaemonSet and configuration. If you configure the cluster such that both theip-masq-agent
DaemonSet is deployed and anonMasqueradeCIDRs
list is specified in theip-masq-agent
ConfigMap, then the--disable-default-snat
flag is no longer relevant. With this DaemonSet and its configuration, non-masquerade destinations are explicitly specified in thenonMasqueradeCIDRs
list, so masquerading is implied for packets sent to destinations not included in thenonMasqueradeCIDRs
list.
Effect of the Egress NAT policy in Autopilot clusters
The GKE Egress NAT policy lets you configure IP masquerading on Autopilot clusters. You can use the GKE Egress NAT policy custom resource definition (CRD) to change the source IP addresses of packets sent from Pods.
For security or IP address exhaustion reasons, you can masquerade IP addresses from Pod to Node IPs for outbound traffic to on-premises networks. For example, you can use a non-RFC-1918 range for Autopilot clusters and use a RFC-1918 range for Nodes. However, if Pods must communicate with on-premises networks that also use non-RFC-1918 range, IP addresses can overlap. To avoid traffic loss, you can configure an Egress NAT Policy to not advertise the Pods' non-RFC-1918 ranges to on-premises networks. The Egress NAT Policy masquerades the Pods' non RFC-1918 range to use the Node's RFC-1918 range instead. Ensure that a Node range is not overlapping with any on-premises range or it can cause a traffic loop.
GKE enforces the IP masquerading behavior for Autopilot clusters through the following process:
- GKE deploys the Egress NAT controller and the
ip-masq-agent
. - You create the Egress NAT policy.
- The GKE controller translates the policy into the
ip-masq-agent
ConfigMap. - The
ip-masq-agent
DaemonSet reads the ConfigMap and then GKE enforces the IP masquerading behavior.
Automatically generated policies
GKE supports two automatically generated Egress NAT policies, default and managed by GKE. Default policies are editable. Managed by GKE policies are fixed and are not editable.
Default policy
GKE predefines a set of default IP address ranges. When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. To change these default IP address ranges, see Edit and deploy the Egress NAT policy.
The following manifest describes a default Egress NAT policy:
Name: default
Namespace:
Labels: <none>
Annotations: <none>
API Version: networking.gke.io/v1
Kind: EgressNATPolicy
Metadata:
Creation Timestamp: 2022-03-16T21:05:45Z
Generation: 2
Managed Fields:
API Version: networking.gke.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:action:
f:status:
Manager: egress-nat-controller
Operation: Update
Time: 2022-03-16T21:05:45Z
API Version: networking.gke.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:destinations:
Manager: kubectl
Operation: Update
Time: 2022-03-17T01:58:13Z
Resource Version: 189346
UID: 06acbb5a-23ba-4c2a-bb34-9b6ed8c4a87f
Spec:
Action: NoSNAT
Destinations:
Cidr: 10.0.0.0/8
Cidr: 172.16.0.0/12
Cidr: 192.168.0.0/16
Cidr: 240.0.0.0/4
Cidr: 192.0.2.0/24
Cidr: 198.51.100.0/24
Cidr: 203.0.113.0/24
Cidr: 100.64.0.0/10
Cidr: 198.18.0.0/15
Cidr: 192.0.0.0/24
Cidr: 192.88.99.0/24
Status:
Events: <none>
The CIDR ranges are the same as the default non-masquerade destination ranges.
Managed by GKE policy
GKE Egress NAT policy reserves a static range of IP addresses required to preserve cluster's operation. This static range contains the Pod, Service, and Node IP address ranges of the cluster, and might overlap with the default policy.
You can identify this policy by a dynamic 8-byte hash (gke-{CLUSTER_SHORT_HASH}
) that GKE assigns. You cannot edit
this policy.
The following manifest describes a managed by GKE policy named gke-bbfa6c0e-1
:
Name: gke-bbfa6c0e-1
Namespace:
Labels: <none>
Annotations: <none>
API Version: networking.gke.io/v1
Kind: EgressNATPolicy
Metadata:
Creation Timestamp: 2022-03-16T21:05:46Z
Generation: 1
Managed Fields:
API Version: networking.gke.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:action:
f:destinations:
f:status:
Manager: egress-nat-controller
Operation: Update
Time: 2022-03-16T21:05:46Z
Resource Version: 11699
UID: 0201b5de-a6f6-4926-822b-31ed7cdee2c6
Spec:
Action: NoSNAT
Destinations:
Cidr: 10.119.128.0/17
Cidr: 10.120.0.0/22
Cidr: 10.128.0.0/20
Status:
Events: <none>
Masquerading to link-local destinations
The link-local IP address destinations fit in the 169.254.0.0/16
IP range. By
default, GKE preserves source Pod IP addresses for packets sent
to 169.254.0.0/16
. This default configuration applies to all
GKE versions regardless of whether the following conditions are met:
- The
ip-masq-agent
DaemonSet is deployed. - The
nonMasqueradeCIDRs
list is specified in theip-masq-agent
ConfigMap. - The
--disable-default-snat
flag is specified when creating the cluster.
You can configure GKE to change source Pod IP addresses to node
IP addresses when sending packets to 169.254.0.0/16
if you deploy the ip-masq-agent
DaemonSet and
set masqLinkLocal
to True in
the ip-masq-agent
ConfigMap. When masqLinkLocal
is not set,
GKE assumes masqLinkLocal
is False.
What's next
- Configure and deploy an IP masquerade agent.
- Read the GKE network overview.
- Learn about configuring authorized networks.