IP addresses and ports

This page describes how Cloud NAT gateways use external IP addresses and how they allocate source ports to Compute Engine virtual machine (VM) instances and Google Kubernetes Engine (GKE) nodes that use the gateways.

Before reviewing this information, familiarize yourself with the Cloud NAT overview.

NAT IP addresses

A NAT IP address is a regional external IP address, routable on the internet. A VM without an external IP address, in a subnetwork (subnet) served by a Cloud NAT gateway, uses a NAT IP address when it sends packets to a destination on the internet.

To assign NAT IP addresses to a Cloud NAT gateway, use one of the following methods:

• Automatic NAT IP address allocation. When you select this option, or choose Google Cloud defaults, Cloud NAT automatically adds regional external IP addresses to your gateway based on Network Service Tiers that you select, the number of VMs that use the gateway, and the number of ports reserved for each VM. It also automatically removes a NAT IP address when it no longer needs any source ports on that NAT IP address.

  • When a Cloud NAT gateway adds a NAT IP address, it creates a static (reserved) regional external IP address in the network tier that you have selected at the time of configuring the gateway. For example, if you have selected Premium Tier, then the Cloud NAT gateway creates the IP address in that tier. The supported network tiers are Premium Tier (default option) and Standard Tier.

    The automatically added NAT IP addresses can be viewed in the list of static external IP addresses. These addresses count towards per-project quotas.

  • If you change the network tier of a Cloud NAT gateway, the existing IP addresses for that gateway are released and a new set of IP addresses from the selected tier is assigned.

  • With automatic allocation, you cannot predict the next IP address that is allocated. If you depend on knowing the set of possible NAT IP addresses ahead of time (for example, to create an allowlist), you should use manual NAT IP address assignment instead.

  • When automatically-added NAT IP addresses are no longer in use, they are removed. However, Cloud NAT only deallocates an address when the last VM assigned to it is no longer using any ports. As such, when the number of VMs using Cloud NAT goes down, you might not see a reduction in IP addresses because Cloud NAT does not dynamically reallocate VMs from one IP address to another because reallocation would disrupt established connections. As long as at least one VM is using an IP address, the IP address stays active and new VMs can be assigned to it.

    If you want to be able to manually reallocate VMs from one IP address to another in order to minimize IP address usage, use manual NAT IP address assignments. This allows manual draining of NAT IP addresses.

  • If you switch to manual NAT IP address assignment later, the automatically reserved regional external IP addresses are deleted. For more information, see Switching assignment method.

• Manual NAT IP address assignment. When you select this option, you create and manually assign static (reserved) regional external IP addresses to your Cloud NAT gateway. You can manually assign IP addresses either from Premium Tier, from Standard Tier, or from both, subject to conditions.

  • You can increase or decrease the number of manually assigned NAT IP addresses by editing the Cloud NAT gateway.
  • When using manual NAT IP address assignment, you must calculate the number of regional external IP addresses that you need for the Cloud NAT gateway. If your gateway runs out of NAT IP addresses, Cloud NAT drops packets. Dropped packets are logged when you use Cloud NAT logging to turn on error logging.
  • For example calculations, see the port reservation example.

For the maximum number of automatically allocated or manually assigned NAT IP addresses, see Cloud NAT limits.

Assign a mix of Premium Tier and Standard Tier IP addresses manually

When you create a manual mode Cloud NAT gateway, you can assign a mix of Premium Tier and Standard Tier IP addresses as long as the IP addresses of different network tiers do not belong to the same rule (including the default rule).

Inside a rule (including the default rule), all IP addresses assigned to active ranges must be of the same network tier. If you try to use IP addresses of different tiers as part of the same rule, Google Cloud rejects the configuration.

Switch assignment method

You can switch a Cloud NAT gateway from automatic NAT IP address allocation to manual NAT IP address assignment; however, the NAT IP addresses cannot be preserved. Even though automatically allocated NAT IP addresses are static, they cannot be moved to a manual NAT IP address assignment. For example, you cannot start using a Cloud NAT gateway with automatically allocated NAT IP addresses and later use those same addresses when you switch the NAT gateway to manually assigned NAT IP addresses.

The set of regional external IP addresses that Cloud NAT uses for automatic NAT IP address allocation are different from the set of regional external IP addresses that you can manually choose.

Drain NAT IP addresses

When you configure a Cloud NAT gateway with manual NAT IP address assignment, you can choose what happens when you need to reduce the number of NAT IP addresses that the gateway uses:

• If you remove a manually assigned NAT IP address, established NAT connections are broken immediately.

• You can choose to drain a manually assigned NAT IP address instead. Draining instructs the Cloud NAT gateway to stop using the NAT IP address for new connections, but continue using it for established connections. Established connections are permitted to close normally instead of being abruptly terminated. To drain an IP address that is associated with a NAT gateway that does not use NAT rules, see Drain external IP addresses associated with NAT. To drain an IP address that is associated with a NAT gateway that uses NAT rules, see Update NAT rules.

Note: You cannot add a new, manually assigned NAT IP address without marking it as immediately eligible for use by the Cloud NAT gateway. Draining is only applicable when removing manually assigned NAT IP addresses.

Ports

Each NAT IP address on a Cloud NAT gateway offers 64,512 TCP source ports and 64,512 UDP source ports. TCP and UDP each support 65,536 ports per IP address, but Cloud NAT doesn't use the first 1,024 well-known (privileged) ports.

When a Cloud NAT gateway performs source network address translation (SNAT) on a packet sent by a VM, it changes the packet's NAT source IP address and source port.

When you create a Cloud NAT gateway, you choose whether to use static port allocation or dynamic port assignment.

Static port allocation

When you configure static port allocation, you specify a minimum number of ports per VM instance.

Because all VMs are allocated the same number of ports, static port allocation works best if all VMs have similar internet usage. If some VMs use more ports than others, the ports in the Cloud NAT gateway might be underused. If internet usage varies, consider configuring dynamic port allocation.

If you want to configure Endpoint-Independent Mapping on your gateway, you must use static port allocation.

Dynamic port allocation

When you configure dynamic port allocation, you specify a minimum number of ports per VM instance and a maximum number of ports per VM instance.

Configuring dynamic port allocation lets the same Cloud NAT gateway allocate different numbers of ports per VM, based on the VM's usage. Initially, a VM is allocated the minimum number of ports per VM instance. If a VM is close to exhausting all the ports that are allocated to it, the number of ports assigned to the VM is doubled. The VM can repeatedly request more ports up to the maximum number of ports per VM instance. When the port usage significantly decreases, the ports are deallocated and can be allocated to other VMs that use the same NAT gateway.

Dynamic port allocation has the following benefits:

• The number of ports that are allocated but not being used are reduced.

• The NAT gateway monitors each VM's port usage and modifies the number of ports allocated to each VM, based on need. You don't need to monitor the port usage or adjust the NAT gateway configuration.

Before using dynamic port allocation, consider the following:

• You cannot configure dynamic port allocation if Endpoint-Independent Mapping is enabled on the Cloud NAT gateway. If you need Endpoint-Independent Mapping, use static port allocation.

• While additional ports are being allocated to VMs, you might see connection timeouts. For strategies to help prevent connections drops, see Reduce dropped connections with dynamic port allocation

Port reservation procedure

Cloud NAT uses this procedure to provision NAT source IP address and source port tuples for each VM that the Cloud NAT gateway serves.

Note: When you use a Cloud NAT gateway to provide NAT services for a private Google Kubernetes Engine cluster, the NAT source IP address and source port tuples are assigned to the node VM, shared by all Pods on the node.

1. Cloud NAT determines the VM internal IP addresses for which NAT should be performed. The VM internal IP addresses are determined by the subnet IP address ranges that the gateway has been configured to serve.

   • If the Cloud NAT gateway is configured to perform NAT for the primary IP address range of the subnet used by the VM's network interface, the gateway performs NAT for both the VM's primary internal IP address and any of the VM's alias IP ranges from the subnet's primary IP address range.

   • If the Cloud NAT gateway is configured to perform NAT for a secondary IP address range of the subnet used by the VM's network interface, the gateway performs NAT for any alias IP ranges from that subnet's secondary IP address range.

2. Cloud NAT adjusts the minimum ports per VM instance if necessary. If static port allocation is configured, and the gateway performs NAT for alias IP ranges that have more than one address (netmask smaller than /32), Cloud NAT adjusts the minimum ports per VM to be the maximum of these two values:

   • The minimum ports per VM instance that you specify

   • The number 1,024

   In all other situations, including when dynamic port allocation is configured, the Cloud NAT gateway proceeds to the next step by using the specified minimum ports per VM instance as input. If the minimum ports per VM instance is not specified, the default value is used; 64 for static port allocation and 32 for dynamic port allocation.

3. Cloud NAT reserves NAT source IP address and source port tuples for each VM. The Cloud NAT gateway uses the given or adjusted minimum ports per VM instance from the previous step to calculate the number of NAT source IP address and source port tuples to assign to the VM.

   Cloud NAT allocates NAT source IP address and source port tuples using multiples of powers of two, such that the number of NAT source IP address and source port tuples is greater than or equal to the minimum ports per VM instance that you specify.

   • It is possible for the NAT source IP address and source port tuples to span more than one NAT IP address if the Cloud NAT gateway uses two or more NAT IP addresses. A single NAT IP address might not have enough available source ports to accommodate the number of NAT source IP address and source port tuples that a VM needs.

   • The Cloud NAT gateway allocates source IP address and source port tuples to each VM.

     • If you have configured static port allocation, the number of source IP addresses and source port tuples is fixed. Each VM can use no more than its allocated number of source IP address and source port tuples, even during traffic bursts.

     • If you have configured dynamic port allocation, the number of source IP addresses and source port tuples can change based on demand. If a VM is close to exhausting its current port allocation, the Cloud NAT allocates additional ports, up to the specified maximum ports per VM instance value. After the VM's port usage reduces below a threshold, the ports are released and can be allocated to other VMs.

Increase ports per VM

If you have configured a Cloud NAT gateway with static port allocation, when you increase the minimum ports per VM on the gateway, there is no interruption to traffic.

If you have configured a Cloud NAT gateway with dynamic port allocation, making further configuration changes might interrupt traffic. When a configuration change is made, the number of ports currently allocated to each VM might be temporarily reset to the minimum number configured.

With both static and dynamic port allocation, you must ensure that the gateway has a sufficient number of NAT IP addresses if you're using manual NAT IP address assignment:

• When using manual NAT IP address assignment, you must calculate the number of NAT source IP addresses that you need. Before increasing the minimum ports per VM, assign at least that many NAT IP addresses to the Cloud NAT gateway.

• When using automatic NAT IP address allocation, increasing the minimum number of ports per VM causes the Cloud NAT gateway to acquire and allocate more regional external IP addresses automatically.

Reduce ports per VM

If you have configured a Cloud NAT gateway with static port allocation, and you reduce the minimum ports per VM on the gateway, there is no connection draining. Established NAT connections are broken immediately and clients must establish new TCP connections.

If you have configured a Cloud NAT gateway with dynamic port allocation, making further configuration changes might interrupt traffic. When a configuration change is made, the number of ports currently allocated to each VM might be temporarily reset to the minimum number configured.

Ports and connections

The number of NAT source IP address and source port tuples that a Cloud NAT gateway reserves for a VM limits the number of connections that the VM can make to a unique destination:

• A unique destination means a unique 3-tuple consisting of a destination IP address, a destination port, and an IP protocol (such as TCP or UDP).

• A connection means a unique 5-tuple consisting of the NAT source IP address and source port tuple combined with a unique destination 3-tuple. Because the UDP protocol is connectionless, the concept of connection is reduced to a 5-tuple associated with a unique UDP datagram.

Suppose that a Cloud NAT gateway calculates 1,024 for the fixed number of ports for a VM by following the port reservation procedure. The Cloud NAT gateway reserves 1,024 unique combinations of NAT source IP address and source port tuples for the VM. The Cloud NAT gateway can process 1,024 simultaneous connections to each unique destination 3-tuple. However, Cloud NAT considers closed connections to be unusable for 120 seconds after the connection closes, which can affect the number of connections in use at a time.

Examples:

• The gateway supports 1,024 simultaneous connections to destination IP address 203.0.113.99 on port 80 using the TCP protocol.

• The gateway supports another 1,024 simultaneous connections to that same destination IP address on port 443, also using the TCP protocol.

• The gateway supports another 1,024 simultaneous connections to a different destination IP address on port 80, also using the TCP protocol.

Simultaneous port reuse and Endpoint-Independent Mapping

Note: The information in this section only applies if you have Endpoint-Independent Mapping enabled.

As long as at least one piece of information in the destination 3-tuple changes—the destination IP address, the destination port, the protocol—the same NAT source IP address and source port tuple can be simultaneously used for many different connections.

Because Cloud NAT uses Endpoint-Independent Mapping, as defined in Section 2.3 of RFC 5128, the number of simultaneous connections that a client VM can make to a unique destination 3-tuple might be reduced if Cloud NAT assigns the same NAT source IP address and source port tuple to more than one internal IP address and ephemeral source port of a client VM. The chances of this happening increase if the client VM has a large number of internal source IP addresses and makes a large number of connections to the same destination 3-tuple. The first time a client VM sends a packet from an internal IP address and ephemeral source port, Cloud NAT creates a many-to-one Endpoint-Independent Mapping between the following:

• The internal IP address and ephemeral source port tuple
• A unique NAT source IP address and source port tuple

For example, when a client VM sends a packet from its internal IP address 10.0.0.2 by using ephemeral source port 10001, Cloud NAT assigns 10.0.0.2:10001 a NAT source IP address and source port tuple to be used for all subsequent connections from 10.0.0.2:10001 to any destination 3-tuple.

If the same VM uses a different ephemeral source port to send a packet, for example, 10.0.0.2:20002, Cloud NAT also assigns a NAT source IP address and source port tuple for all subsequent connections from 10.0.0.2:20002 to any destination 3-tuple. It is possible that Cloud NAT could assign the same NAT source IP address and source port tuple to both of these internal IP address and ephemeral source port tuples. In certain situations, this causes an endpoint independent conflict.

For a more detailed example, see Endpoint-Independent Mapping conflict example.

Reduce endpoint independent conflicts

You can make configuration changes to reduce endpoint independent conflicts. For more information, see Packets dropped with reason endpoint independent conflict.

Delay for TCP source port reuse

After a Cloud NAT gateway closes a TCP connection, Google Cloud enforces a delay before the gateway can reuse the same NAT source IP address and source port tuple with the same destination (destination IP address, destination port, and protocol). The length of the delay is controlled by the TCP TIME_WAIT Timeout setting.

If needed, you can reduce this delay or you can make one of the following changes:

• Increase the minimum number of ports per VM instance so that the port reservation procedure assigns the VM more NAT source IP address and source port tuples.

• If a VM needs to rapidly open and close TCP connections to the same destination IP address and destination port by using the same protocol, you should assign an external IP address to the VM and use firewall rules to limit unsolicited ingress connections instead of using Cloud NAT.

Source ports and security

If you depend on source port randomization as a security measure, you need to consider the following:

• Increase the minimum number of ports per VM instance so that the port reservation procedure assigns the VM more NAT source IP address and source port tuples. This assigns a range of ports randomly to each VM; however, the source port chosen from that range is sequential.

• Assign an external IP address to the VM instead of using Cloud NAT.

Examples

The following examples demonstrate how Cloud NAT reserves NAT source IP addresses and source ports for a VM and how it performs NAT for packets sent to the internet.

Port reservation

The following examples demonstrate applications of the port reservation procedure.

Suppose you're configuring a Cloud NAT gateway to provide NAT for the primary IP address range of a subnet, and the VMs that use that subnet do not have any alias IP ranges from the subnet's primary IP address range. Round down the result of any division operation to the closest integer. ⌊⌋ is the floor (greatest integer) function, meaning discard any fractional result of division.

• If you configure the Cloud NAT gateway with a single NAT IP address using manual assignment, and you set the minimum number of ports per VM instance to 64, the gateway can provide NAT services for up to 1,008 VMs:

  ⌊(1 NAT IP address) × (64,512 ports per address) / (64 ports per VM)⌋ = 1,008 VMs

• If you need to support more than 1,008 VMs, you can assign a second NAT IP address to the Cloud NAT gateway. With two NAT IP addresses, keeping the minimum number of ports per VM at 64, you can support 2,016 VMs:

  ⌊(2 NAT IP addresses) × (64,512 ports per address) / (64 ports per VM)⌋ = 2,016 VMs

• If you set the minimum number of ports per VM to 4,096, each NAT IP address can support 15 VMs. This calculation is rounded down to the closest integer:

  ⌊(1 NAT IP addresses) × (64,512 ports per address) / (4,096 ports per VM)⌋ = 15 VMs

Endpoint-Independent Mapping conflict

Note: The information in this section only applies if you have Endpoint-Independent Mapping enabled.

The following example illustrates how Endpoint-Independent Mapping might reduce the number of simultaneous connections from a client VM to the same destination 3-tuple, even when there is a sufficient number of free NAT source IP address and source port tuples for the client VM.

Suppose you've configured a Cloud NAT gateway to provide NAT for the primary IP address range of a subnet. You've created a client VM with one network interface whose primary internal IP address is 10.0.0.2 in that subnet. The example VM does not have an external IP address assigned to its network interface.

1. The VM opens a connection with these characteristics:

   • Source internal IP address and port: 10.0.0.2:10001
   • Destination 3-tuple: 203.0.113.1:80 using TCP
   • Cloud NAT uses the following NAT source IP address and source port tuple: 192.0.2.10:30009

2. The VM opens a second connection with these characteristics:

   • Source internal IP address and port: 10.0.0.2:10002
   • Destination 3-tuple: 203.0.113.2:80 using TCP
   • Cloud NAT might choose to use the same NAT source IP address and source port tuple,192.0.2.10:30009, for this connection as well. Using the same NAT source IP address and source port tuple for a different client IP address and ephemeral source port is possible.

3. While both the first and the second connections are active, Cloud NAT cannot open a third TCP connection with these characteristics:

   • Same source internal IP address and port as the first connection: 10.0.0.2:10001
   • Same destination 3-tuple as the second connection: 203.0.113.2:80 using TCP

   This third connection attempt is dropped with an endpoint independent conflict error because the Endpoint-Independent Mapping established by the first connection mandates that all connections from 10.0.0.2:10001 must use the same NAT source IP address and source port tuple, 192.0.2.10:30009, but 192.0.2.10:30009 is already being used by the second TCP connection to 203.0.113.2:80.

4. To dispel ambiguity, a subsequent connection attempt in this example is successful as long as one of the following is true:

   • The first TCP connection has been closed. This removes the Endpoint-Independent Mapping between 10.0.0.2:10001 and 192.0.2.10:30009, so the third connection can be mapped to a different NAT source IP address and source port tuple to communicate with 203.0.113.2:80 using TCP.
   • The second TCP connection has been closed. This frees up 10.0.0.2:10001 to use the NAT source IP address and source port 192.0.2.10:30009 to communicate with 203.0.113.2:80 using TCP.
   • The third connection attempt selects a different ephemeral (internal) source port. In this example, an Endpoint-Independent Mapping established a many-to-one mapping for internal NAT source IP addresses and source ports 10.0.0.2:10001 and 10.0.0.2:10002 to use 192.0.2.10:30009 when communicating with 203.0.113.2:80 using TCP. If the third connection attempt uses an ephemeral source port different from both 10001 and 10002, there's a chance that a different NAT source IP address and source port can be used to communicate with 203.0.113.2:80 using TCP.
   • Toggling endpoint independence off. This allows the new connection from 10.0.0.2:10001 to not need to use 192.0.2.10:30009, allowing it to use a different NAT source IP address and port.

For techniques that you can use to avoid conflicts, see Reducing endpoint independent conflicts.

NAT flow

In this example, a VM with primary internal IP address 10.240.0.4, without an external IP address, needs to download an update from the external IP address 203.0.113.1. You've configured the nat-gw-us-east gateway as follows:

• Minimum ports per instance: 64
• Manually assigned two NAT IP addresses: 192.0.2.50 and 192.0.2.60.
• Provided NAT for the primary IP address range of subnet-1.

Cloud NAT translation example (click to enlarge)

Cloud NAT follows the port reservation procedure to reserve the following NAT source IP address and source port tuples for each of the VMs in the network. For example, the Cloud NAT gateway reserves 64 source ports for the VM with internal IP address 10.240.0.4. The NAT IP address 192.0.2.50 has 64 unreserved ports, so the gateway reserves the following set of 64 NAT source IP address and source port tuples for that VM:

• 192.0.2.50:34000 through 192.0.2.50:34063

When the VM sends a packet to the update server 203.0.113.1 on destination port 80, using the TCP protocol, the following occurs:

• The VM sends a request packet with these attributes:

  • Source IP address: 10.240.0.4, the primary internal IP address of the VM
  • Source port: 24000, the ephemeral source port chosen by the VM's operating system
  • Destination address: 203.0.113.1, the update server's external IP address
  • Destination port: 80, the destination port for HTTP traffic to the update server
  • Protocol: TCP

• The nat-gw-us-east gateway performs SNAT on egress, rewriting the request packet's NAT source IP address and source port. The modified packet is sent to the internet if the VPC network has a route for the 203.0.113.1 destination whose next hop is the default internet gateway. A default route commonly meets this requirement.

  • NAT source IP address: 192.0.2.50, from one of the VM's reserved NAT source IP address and source port tuples
  • Source port: 34022, an unused source port from one of the VM's reserved source port tuples
  • Destination address: 203.0.113.1, unchanged
  • Destination port: 80, unchanged
  • Protocol: TCP, unchanged

• When the update server sends a response packet, that packet arrives on the nat-gw-us-east gateway with these attributes:

  • Source IP address: 203.0.113.1, the update server's external IP address
  • Source port: 80, the HTTP response from the update server
  • Destination address: 192.0.2.50, matching the original NAT source IP address of the request packet
  • Destination port: 34022, matching the source port of the request packet
  • Protocol: TCP, unchanged

• The nat-gw-us-east gateway performs destination network address translation (DNAT) on the response packet, rewriting the response packet's destination address and destination port so that the packet is delivered to the VM:

  • Source IP address: 203.0.113.1, unchanged
  • Source port: 80, unchanged
  • Destination address: 10.240.0.4, the primary internal IP address of the VM
  • Destination port: 24000, matching the original ephemeral source port of the request packet
  • Protocol: TCP, unchanged

What's next

• Create your own Cloud NAT gateway.
• Create an example Compute Engine setup.
• Create an example GKE setup.
• Troubleshoot common issues.