Cloud NAT address and port overview

This page describes how Cloud NAT gateways use external IP addresses and how they allocate source ports to Compute Engine virtual machine (VM) instances and Google Kubernetes Engine (GKE) nodes that use the gateways.

Before reviewing this information, familiarize yourself with the Cloud NAT overview.

NAT IP addresses

A NAT IP address is a regional external IP address, routable on the internet. A VM without an external IP address, in a subnetwork (subnet) served by a Cloud NAT gateway, uses a NAT IP address when it sends packets to a destination on the internet.

To assign NAT IP addresses to a Cloud NAT gateway, use one of the following methods:

  • Automatic NAT IP address allocation. When you select this option, or choose Google Cloud defaults, Cloud NAT automatically adds regional external IP addresses to your gateway based on the number of VMs that use the gateway and the number of ports reserved for each VM. It also automatically removes a NAT IP address when it no longer needs any source ports on that NAT IP address.

    • When a Cloud NAT gateway adds a NAT IP address, it creates a static (reserved) regional external IP address. These addresses count towards per-project quotas.
    • With automatic allocation, you cannot predict the next IP address that is allocated. If you depend on knowing the set of possible NAT IP addresses ahead of time (for example, to create an allowlist), you should use manual NAT IP address assignment instead.
    • If you switch to manual NAT IP address assignment later, the automatically reserved regional external IP addresses are deleted. For more information, see Switching assignment method.
  • Manual NAT IP address assignment. When you select this option, you create and manually assign static (reserved) regional external IP addresses to your Cloud NAT gateway. You can increase or decrease the number of manually assigned NAT IP addresses by editing the Cloud NAT gateway.

    • When using manual NAT IP address assignment, you must calculate the number of regional external IP addresses that you need for the Cloud NAT gateway. If your gateway runs out of NAT IP addresses, Cloud NAT drops packets. Dropped packets are logged when you use Cloud NAT logging to turn on error logging.
    • For example calculations, see the port reservation example.

For the maximum number of automatically allocated or manually assigned NAT IP addresses, see Cloud NAT limits.

Switching assignment method

You can switch a Cloud NAT gateway from automatic NAT IP address allocation to manual NAT IP address assignment; however, the NAT IP addresses cannot be preserved. Even though automatically allocated NAT IP addresses are static, they cannot be moved to a manual NAT IP address assignment. For example, you cannot start using a Cloud NAT gateway with automatically allocated NAT IP addresses and later use those same addresses when you switch the NAT gateway to manually assigned NAT IP addresses.

The set of regional external IP addresses that Cloud NAT uses for automatic NAT IP address allocation are different from the set of regional external IP addresses that you can manually choose.

Draining NAT IP addresses

When you configure a Cloud NAT gateway with manual NAT IP address assignment, you can choose what happens when you need to reduce the number of NAT IP addresses that the gateway uses:

  • If you remove a manually assigned NAT IP address, established NAT connections are broken immediately.

  • You can choose to drain a manually assigned NAT IP address instead. Draining instructs the Cloud NAT gateway to stop using the NAT IP address for new connections, but continue using it for established connections. Established connections are permitted to close normally instead of being abruptly terminated. For instructions, see Drain external IP addresses associated with NAT.

Ports

Each NAT IP address on a Cloud NAT gateway offers 64,512 TCP source ports and 64,512 UDP source ports. TCP and UDP each support 65,536 ports per IP address, and Cloud NAT doesn't use the first 1,024 well-known (privileged) ports.

When you create a Cloud NAT gateway, you specify a minimum number of ports per VM instance. When a Cloud NAT gateway performs source network address translation (SNAT) on a packet sent by a VM, it changes the packet's NAT source IP address and source port.

Port reservation procedure

Cloud NAT uses this procedure to provision NAT source IP address and source port tuples for each VM that the Cloud NAT gateway serves. When you use a Cloud NAT gateway to provide NAT services for a private Google Kubernetes Engine cluster, the number of NAT source IP address and source port tuples are assigned to the whole node VM, shared by all Pods on the node.

  1. Determine the VM internal IP addresses for which NAT should be performed. The VM internal IP addresses for which a Cloud NAT gateway should perform NAT is determined by the subnet IP address ranges that the gateway has been configured to serve.

    • If the Cloud NAT gateway is configured to perform NAT for the primary IP address range of the subnet used by the VM's network interface, the gateway performs NAT for both the VM's primary internal IP address and any of the VM's alias IP ranges from the subnet's primary IP address range.

    • If the Cloud NAT gateway is configured to perform NAT for a secondary IP address range of the subnet used by the VM's network interface, the gateway performs NAT for any alias IP ranges from that subnet's secondary IP address range.

  2. Calculate the number of source ports to assign to the VM.

    1. If the Cloud NAT gateway should perform NAT for one or more of the VM's alias IP ranges whose netmasks are shorter than /32—that is, if the gateway should perform NAT for at least one of the VM's alias IP ranges that consists of more than a single IP address—Cloud NAT takes the maximum of the following two numbers and proceeds to the next step by using that maximum as input:

      • The minimum ports per VM instance that you specified
      • The number 1,024

      Otherwise, the Cloud NAT gateway proceeds to the next step by using just the minimum ports per VM instance as input.

    2. The Cloud NAT gateway uses this input to determine how many source ports it should assign to the VM.

  1. Reserve NAT source IP address and source port tuples for the VM. The number of NAT source IP address and source port tuples matches how many source ports the Cloud NAT gateway determined should be assigned to the VM.

    It is possible for the NAT source IP address and source port tuples to span more than one NAT IP address if the Cloud NAT gateway uses two or more NAT IP addresses. A single NAT IP address might not have enough unreserved source ports to accommodate the number of NAT source IP address and source port tuples that the VM needs.

Reducing ports

You can reduce the number of Minimum ports per VM on already configured Cloud NAT resources. However, there is no connection draining. When you reduce the number of ports per VM, established NAT connections are broken immediately.

Ports and connections

The number of NAT source IP address and source port tuples that a Cloud NAT gateway reserves for a VM limits the number of connections that the VM can make to a unique destination:

  • A unique destination means a unique 3-tuple consisting of a destination IP address, a destination port, and an IP protocol (such as TCP or UDP).

  • A connection means a unique 5-tuple consisting of the NAT source IP address and source port tuple combined with a unique destination 3-tuple. Because the UDP protocol is connectionless, the concept of connection is reduced to a 5-tuple associated with a unique UDP datagram.

Suppose that a Cloud NAT gateway calculates 1,024 for the fixed number of ports for a VM by following the port reservation procedure. The Cloud NAT gateway reserves 1,024 unique combinations of NAT source IP address and source port tuples for the VM. The Cloud NAT gateway can process 1,024 simultaneous connections to each unique destination 3-tuple. However, Cloud NAT considers closed connections to be unusable for 120 seconds after the connection closes, which can affect the number of connections in use at a time.

Examples:

  • The gateway supports 1,024 simultaneous connections to destination IP address 203.0.113.99 on port 80 using the TCP protocol.

  • The gateway supports another 1,024 simultaneous connections to that same destination IP address on port 443, also using the TCP protocol.

  • The gateway supports another 1,024 simultaneous connections to a different destination IP address on port 80, also using the TCP protocol.

Simultaneous port reuse and Endpoint-Independent Mapping

As long as at least one piece of information in the destination 3-tuple changes—the destination IP address, the destination port, the protocol—the same NAT source IP address and source port tuple can be simultaneously used for many different connections.

Because Cloud NAT uses Endpoint-Independent Mapping, as defined in Section 2.3 of RFC 5128, the number of simultaneous connections that a client VM can make to a unique destination 3-tuple might be reduced if Cloud NAT assigns the same NAT source IP address and source port tuple to more than one internal IP address and ephemeral source port of a client VM. The chances of this happening increase if the client VM has a large number of internal source IP addresses and makes a large number of connections to the same destination 3-tuple. The first time a client VM sends a packet from an internal IP address and ephemeral source port, Cloud NAT creates a many-to-one Endpoint-Independent Mapping between the following:

  • The internal IP address and ephemeral source port tuple
  • A unique NAT source IP address and source port tuple

For example, when a client VM sends a packet from its internal IP address 10.0.0.2 by using ephemeral source port 10001, Cloud NAT assigns 10.0.0.2:10001 a NAT source IP address and source port tuple to be used for all subsequent connections from 10.0.0.2:10001 to any destination 3-tuple.

If the same VM uses a different ephemeral source port to send a packet, for example, 10.0.0.2:20002, Cloud NAT also assigns a NAT source IP address and source port tuple for all subsequent connections from 10.0.0.2:20002 to any destination 3-tuple. It is possible that Cloud NAT could assign the same NAT source IP address and source port tuple to both of these internal IP address and ephemeral source port tuples. In certain situations, this causes an endpoint independent conflict.

For a more detailed example, see Endpoint-Independent Mapping conflict example.

Reducing endpoint independent conflicts

You can reduce the chances of endpoint independent conflicts by using the following techniques:

  • Turn off Endpoint-Independent Mapping.

  • Increase the minimum number of ports per VM instance, so that the port reservation procedure can assign more NAT source IP address and source port tuples to each client VM. This decreases the probability that two or more client IP address and ephemeral source port tuples are assigned the same NAT source IP address and source port tuple.

  • Configure your VM instances to use a larger set of ephemeral source ports:

    • On Linux VMs, you can set the ip_local_port_range to the maximum number of ephemeral source ports (64,512) with this command:

      echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
      
    • On Windows VMs, you can set the number of ephemeral source TCP and UDP ports to the maximum possible (64,512) with these commands:

      netsh int ipv4 set dynamicport tcp start=1024 num=64512
      netsh int ipv4 set dynamicport udp start=1024 num=64512
    • On GKE nodes, you can automate this configuration by using a privileged DaemonSet.

  • For GKE clusters, disable the source NAT performed on each node for packets sent to destinations of interest. You can do this in one of two ways:

Delay for TCP source port reuse

After a Cloud NAT gateway closes a TCP connection, Google Cloud enforces a two-minute delay before the gateway can reuse the same NAT source IP address and source port tuple with the same destination (destination IP address, destination port, and protocol).

You cannot reduce this delay; however, you can do one of the following:

  • Increase the minimum number of ports per VM instance so that the port reservation procedure assigns the VM more NAT source IP address and source port tuples.

  • If a VM needs to rapidly open and close TCP connections to the same destination IP address and destination port by using the same protocol, you should assign an external IP address to the VM and use firewall rules to limit unsolicited ingress connections instead of using Cloud NAT.

Source ports and security

If you depend on source port randomization as a security measure, you need to consider the following:

Examples

The following examples demonstrate how Cloud NAT reserves NAT source IP addresses and source ports for a VM and how it performs NAT for packets sent to the internet.

Port reservation

The following examples demonstrate applications of the port reservation procedure.

Suppose you're configuring a Cloud NAT gateway to provide NAT for the primary IP address range of a subnet, and the VMs that use that subnet do not have any alias IP ranges from the subnet's primary IP address range. Round down the result of any division operation to the closest integer. ⌊⌋ is the floor (greatest integer) function, meaning discard any fractional result of division.

  • If you configure the Cloud NAT gateway with a single NAT IP address using manual assignment, and you set the minimum number of ports per VM instance to 64, the gateway can provide NAT services for up to 1,008 VMs:

    ⌊(1 NAT IP address) × (64,512 ports per address) / (64 ports per VM)⌋ = 1,008 VMs

  • If you need to support more than 1,008 VMs, you can assign a second NAT IP address to the Cloud NAT gateway. With two NAT IP addresses, keeping the minimum number of ports per VM at 64, you can support 2,016 VMs:

    ⌊(2 NAT IP addresses) × (64,512 ports per address) / (64 ports per VM)⌋ = 2,016 VMs

  • If you set the minimum number of ports per VM to 4,096, each NAT IP address can support 15 VMs. This calculation is rounded down to the closest integer:

    ⌊(1 NAT IP addresses) × (64,512 ports per address) / (4,096 ports per VM)⌋ = 15 VMs

Endpoint-Independent Mapping conflict

The following example illustrates how Endpoint-Independent Mapping might reduce the number of simultaneous connections from a client VM to the same destination 3-tuple, even when there is a sufficient number of free NAT source IP address and source port tuples for the client VM.

Suppose you've configured a Cloud NAT gateway to provide NAT for the primary IP address range of a subnet. You've created a client VM with one network interface whose primary internal IP address is 10.0.0.2 in that subnet. The example VM does not have an external IP address assigned to its network interface.

  1. The VM opens a connection with these characteristics:

    • Source internal IP address and port: 10.0.0.2:10001
    • Destination three-tuple: 203.0.113.1:80 using TCP
    • Cloud NAT uses the following NAT source IP address and source port tuple: 192.0.2.10:30009
  2. The VM opens a second connection with these characteristics:

    • Source internal IP address and port: 10.0.0.2:10002
    • Destination three-tuple: 203.0.113.2:80 using TCP
    • Cloud NAT might choose to use the same NAT source IP address and source port tuple,192.0.2.10:30009, for this connection as well. Using the same NAT source IP address and source port tuple for a different client IP address and ephemeral source port is possible.
  3. While both the first and the second connections are active, Cloud NAT cannot open a third TCP connection with these characteristics:

    • Same source internal IP address and port as the first connection: 10.0.0.2:10001
    • Same destination three-tuple as the second connection: 203.0.113.2:80 using TCP

    This third connection attempt is dropped with an endpoint independent conflict error because the Endpoint-Independent Mapping established by the first connection mandates that all connections from 10.0.0.2:10001 must use the same NAT source IP address and source port tuple, 192.0.2.10:30009, but 192.0.2.10:30009 is already being used by the second TCP connection to 203.0.113.2:80.

  4. To dispel ambiguity, a subsequent connection attempt in this example is successful as long as one of the following is true:

    • The first TCP connection has been closed. This removes the Endpoint-Independent Mapping between 10.0.0.2:10001 and 192.0.2.10:30009, so the third connection can be mapped to a different NAT source IP address and source port tuple to communicate with 203.0.113.2:80 using TCP.
    • The second TCP connection has been closed. This frees up 10.0.0.2:10001 to use the NAT source IP address and source port 192.0.2.10:30009 to communicate with 203.0.113.2:80 using TCP.
    • The third connection attempt selects a different ephemeral (internal) source port. In this example, an endpoint independent mapping established a many-to-one mapping for internal NAT source IP addresses and source ports 10.0.0.2:10001 and 10.0.0.2:10002 to use 192.0.2.10:30009 when communicating with 203.0.113.2:80 using TCP. If the third connection attempt uses an ephemeral source port different from both 10001 and 10002, there's a chance that a different NAT source IP address and source port can be used to communicate with 203.0.113.2:80 using TCP.

For techniques that you can use to avoid conflicts, see Reducing endpoint independent conflicts.

NAT flow

In this example, a VM with primary internal IP address 10.240.0.4, without an external IP address, needs to download an update from the external IP address 203.0.113.1. You've configured the nat-gw-us-east gateway as follows:

  • Minimum ports per instance: 64
  • Manually assigned two NAT IP addresses: 192.0.2.50 and 192.0.2.60.
  • Provided NAT for the primary IP address range of subnet-1.
Cloud NAT translation example (click to enlarge).
Cloud NAT translation example (click to enlarge)

Cloud NAT follows the port reservation procedure to reserve the following NAT source IP address and source port tuples for each of the VMs in the network. For example, the Cloud NAT gateway reserves 64 source ports for the VM with internal IP address 10.240.0.4. The NAT IP address 192.0.2.50 has 64 unreserved ports, so the gateway reserves the following set of 64 NAT source IP address and source port tuples for that VM:

  • 192.0.2.50:34000 through 192.0.2.50:34063

When the VM sends a packet to the update server 203.0.113.1 on destination port 80, using the TCP protocol, the following occurs:

  • The VM sends a request packet with these attributes:

    • NAT source IP address: 10.240.0.4, the primary internal IP address of the VM
    • Source port: 24000, the ephemeral source port chosen by the VM's operating system
    • Destination address: 203.0.113.1, the update server's external IP address
    • Destination port: 80, the destination port for HTTP traffic to the update server
    • Protocol: TCP
  • The nat-gw-us-east gateway performs SNAT on egress, rewriting the request packet's NAT source IP address and source port. The modified packet is sent to the internet if the VPC network has a route for the 203.0.113.1 destination whose next hop is the default internet gateway. A default route commonly meets this requirement.

    • NAT source IP address: 192.0.2.50, from one of the VM's reserved NAT source IP address and source port tuples
    • Source port: 34022, an unused source port from one of the VM's reserved source port tuples
    • Destination address: 203.0.113.1, unchanged
    • Destination port: 80, unchanged
    • Protocol: TCP, unchanged
  • When the update server sends a response packet, that packet arrives on the nat-gw-us-east gateway with these attributes:

    • NAT source IP address: 203.0.113.1, the update server's external IP address
    • Source port: 80, the HTTP response from the update server
    • Destination address: 192.0.2.50, matching the original NAT source IP address of the request packet
    • Destination port: 34022, matching the source port of the request packet
    • Protocol: TCP, unchanged
  • The nat-gw-us-east gateway performs destination network address translation (DNAT) on the response packet, rewriting the response packet's destination address and destination port so that the packet is delivered to the VM:

    • NAT source IP address: 203.0.113.1, unchanged
    • Source port: 80, unchanged
    • Destination address: 10.240.0.4, the primary internal IP address of the VM
    • Destination port: 24000, matching the original ephemeral source port of the request packet
    • Protocol: TCP, unchanged

What's next