This guide can help you solve common issues with Cloud NAT.
VMs can reach the internet unexpectedly, without Cloud NAT
If your virtual machine (VM) instances or container instances can reach the internet without Cloud NAT, but you don't want them to, check for the following issues:
Determine if the VM's network interface has an external IP address. If the network interface has an external IP address assigned to it, Google Cloud automatically performs one-to-one NAT for packets whose sources match the interface's primary internal IP address. For more information, see Cloud NAT specifications.
To determine if a VM has an external IP address, see changing or assigning an external IP address to an existing instance.
Ensure that your Google Kubernetes Engine (GKE) cluster is a private cluster. Each node VM in a non-private cluster has an external IP address, so each node can use routes in your Virtual Private Cloud (VPC) network whose next hop is the default internet gateway without relying on Cloud NAT. For more information, including how non-private clusters interact with Cloud NAT gateways, see GKE interaction.
List routes in your VPC network, looking for ones that could provide internet connectivity through a next hop different than the default internet gateway. As examples:
Custom static routes whose next hops are VMs, internal TCP/UDP load balancers, or Cloud VPN tunnels might indirectly provide internet connectivity. For example, the next hop VMs or backend VMs for an internal TCP/UDP load balancer might have external IP addresses themselves, or a Cloud VPN tunnel might connect to a network that offers internet access.
Custom dynamic routes learned from on-premises networks by Cloud Routers in your VPC network might connect to a network that offers internet access.
Keep in mind that other custom routes in your VPC network might have higher priorities than routes whose next hops are default internet gateways. For information about how Google Cloud evaluates routes, see routing applicability and order.
No logs are generated
- Verify that NAT logging is enabled.
Double-check that your view of the logs isn't filtering out the logs that you are looking for. For instructions, see Viewing logs.
Make sure that a firewall rule isn't blocking traffic. Firewall rules that block egress (outbound) traffic are applied before the traffic would have been sent to the NAT gateway. You can use Firewall Rules Logging to see if your custom egress rules are blocking outbound traffic.
Review cases where NAT is not performed on traffic. The destination for your traffic might not be handled by NAT.
Certain logs are excluded
Cloud NAT does not log every single event. During periods of heavy egress traffic, NAT logging is throttled, proportional to the machine type of the VM. Translation or error logs might be dropped, and it is not possible to determine what is omitted during throttling.
Packets dropped with reason: out of resources
If you see packet loss from VMs that use Cloud NAT, this might be because there are not enough available NAT source IP address and source port tuples for the VM to use at the time of the packet loss (port exhaustion). A five-tuple (NAT source IP address, source port, and destination 3-tuple) cannot be reused within the mandatory 120-second delay.
To increase the number of ports per VM, see Change minimum default number of NAT ports allocated per VM associated with NAT.
Packets dropped with reason: endpoint independent conflict
If you see packet loss from VMs that use Cloud NAT, and you have
Endpoint-Independent Mapping turned on, the packet loss might be caused by an
conflict. If it is, the
ENDPOINT_INDEPENDENT_CONFLICT. For more information about metrics, see Using
VM instance metrics.
You can reduce the chances of endpoint independent conflicts by using the following techniques:
Turn off Endpoint-Independent Mapping. This allows the new connection from a given source IP address and port to use a different NAT source IP address and port than it used before. Disabling or enabling Endpoint-Independent Mapping does not interrupt established connections.
Increase the minimum default number of NAT ports per VM instance, so that the port reservation procedure can assign more NAT source IP address and source port tuples to each client VM. This decreases the probability that two or more client IP address and ephemeral source port tuples are assigned the same NAT source IP address and source port tuple.
Configure your VM instances to use a larger set of ephemeral source ports:
For Linux VMs:
You can view what port range is configured with this command:
You can set the
ip_local_port_rangeto the maximum number of ephemeral source ports (64,512) with this command:
echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
For Windows VMs:
You can view what port ranges are configured with these commands:
netsh int ipv4 show dynamicport tcp netsh int ipv4 show dynamicport udp
You can set the number of ephemeral source TCP and UDP ports to the maximum possible (64,512) with these commands:
netsh int ipv4 set dynamicport tcp start=1024 num=64512 netsh int ipv4 set dynamicport udp start=1024 num=64512
On GKE nodes, you can automate this configuration by using a privileged
For GKE clusters, disable the source NAT performed on each node for packets sent to destinations of interest. You can do this in one of two ways:
Need to allocate more IP addresses
If an instance is unable to reach the internet, and if you are manually allocating IP addresses, you might need to add more IP addresses.
In the Cloud Console, if you see the message You need to allocate at least 'X' more IP addresses to allow all instances to access the internet, then you need to allocate more IP addresses. For more information, see NAT IP addresses.
Frequently asked questions
Regional restriction for Cloud NAT
Can I use the same Cloud NAT gateway in more than one region?
No. Cloud NAT gateways are regional resources, associated with a single region, VPC network, and Cloud Router.
You can create additional Cloud NAT gateways in other regions or other VPC networks. To determine if you can create more than one gateway in a given region and VPC network, see subnet IP address range applicability.
Are the external NAT IP addresses used by Cloud NAT gateways global or regional?
Cloud NAT gateways use regional external IP addresses as NAT IP addresses. Even though they are regional, they are publicly routable. For information about different ways that NAT IP addresses can be allocated or assigned, see NAT IP addresses.
When Cloud NAT can and cannot be used
Does Cloud NAT apply to instances, including GKE node VMs, that have external IP addresses?
Generally, no. If the network interface of a VM has an external IP address, Google Cloud always performs 1-to-1 NAT for packets sent from the primary internal IP address of the network interface without using Cloud NAT. However, Cloud NAT could still provide NAT services to packets sent from alias IP address ranges of that same network interface. For additional details, see Cloud NAT specifications and GKE interaction.
Can I use Cloud NAT for communication between VMs in a VPC network?
No, Cloud NAT is designed to provide connectivity to the internet only.
Can I use Cloud NAT to connect a VPC network to another network to work around overlapping IP addresses?
No, Cloud NAT cannot apply to any custom route whose next hop is not the default internet gateway. For example, Cloud NAT cannot apply to traffic sent to a next hop Cloud VPN tunnel, even if the destination is a publicly routable IP address.
Does Cloud NAT let a source VM whose network interface lacks an external IP address send traffic to a destination VM or load balancer that has an external IP address, even when the source and destination are in the same VPC network?
Yes. The network path involves sending traffic out of the VPC network through a default internet gateway, and then receiving it in the same network.
When the source VM sends a packet to the destination, Cloud NAT performs source NAT (SNAT) before delivering the packet to the second instance. Cloud NAT performs destination NAT (DNAT) for responses from the second instance to the first. For a step-by-step example, see NAT flow.
Unsolicited incoming connections not supported
Does Cloud NAT allow for inbound connections (for example, SSH) to instances without external IP addresses?
No, Cloud NAT does not support unsolicited incoming connections. For more information, see Cloud NAT specifications.
If you need to connect to a VM that doesn't have an external IP address, see connecting to instances that do not have external IP addresses. For example, as part of the Cloud NAT example Compute Engine setup, you connect to a VM without an external IP address by using Identity-Aware Proxy.
Cloud NAT and ports
Why does a VM have a fixed number of ports (
64 by default)?
When a Cloud NAT gateway provides NAT for a VM, it reserves source address and source port tuples according to the port reservation procedure.
For more information, see port reservation examples.
Can I change the minimum number of ports reserved for a VM?
Yes. You can increase or decrease the minimum number of ports per VM when you create a new Cloud NAT gateway or by editing it later. Each Cloud NAT gateway reserves source address and source port tuples according to the port reservation procedure.
For additional information about decreasing the minimum number of ports, see the next question.
Can I decrease the minimum number of ports per VM after creating the Cloud NAT gateway?
Yes; however, decreasing the minimum number of ports could result in the port reservation procedure reserving a smaller number of ports per VM. When this happens, existing TCP connections might be reset and, if so, must be re-established.
When switching NAT mapping from Primary and Secondary ranges to Primary range only, are additional ports allocated to each instance immediately released?
No. Any additional ports used by secondary ranges are retained by instances until the minimum ports per VM setting is reduced. When Cloud NAT is configured to map Secondary (alias) ranges for subnets, Cloud NAT assigns a minimum of 1,024 ports per instance, based on the port reservation procedure.
By switching to Primary ranges only, Cloud NAT conserves those additional allocated ports for instances that have already had those ports assigned. After changing the ranges for which Cloud NAT is applied to Primary only, the actual number of ports assigned to those instances is not changed until the minimum ports per VM setting is also reduced.
To reduce the amount of ports allocated to those instances, after switching to primary ranges, the minimum ports per VM setting must be reduced. After that value is reduced, Cloud NAT automatically adjusts the number of ports allocated per instance down, which reduces port consumption.