Network bandwidth

Google Cloud accounts for bandwidth per virtual machine (VM) instance, not per network interface (NIC) or IP address. A VM's machine type defines its maximum possible egress rate; however, you can only achieve that maximum possible egress rate in specific situations.

This page outlines expectations, which are useful when planning your deployments. It categorizes bandwidth using two dimensions:

The traffic direction: As used on this page, traffic direction is always from the perspective of a Google Cloud VM:
- Packets sent from a Google Cloud VM compose its egress (outbound) traffic.
- Packets sent to a Google Cloud VM compose its ingress (inbound) traffic.
The type of destination IP address: Google Cloud categorizes IP addresses as either internal or external:
- An IP address within a VPC network is called an internal IP address. For example, each VM's NIC has a primary internal IP address located in a VPC network. Internal IP addresses can be any valid private or privately re-used public IP address range.
- An IP address that's accessible from the internet is an external IP address. External IP addresses can be located in Google Cloud, such as the external IP address assigned to a VM's NIC. External IP addresses are always public IP addresses, including public IP addresses on the internet outside of Google Cloud.
See IP addresses in the VPC documentation for precise definitions. For example, if you privately re-use a public IP address in your VPC network, that is an internal IP address, and the corresponding external IP address is no longer reachable.

All of the information on this page is applicable to Compute Engine VMs, as well as products that depend on Compute Engine VMs. For example, a Google Kubernetes Engine node is a Compute Engine VM.

Neither additional network interfaces (NICs) nor additional IP addresses per NIC increase ingress or egress bandwidth for a VM. For example, an n1-standard-8 VM with two NICs is limited to 16 Gbps total egress bandwidth, not 16 Gbps egress bandwidth per NIC.

Bandwidth summary table

The following table summarizes bandwidth expectations:

	Packet's destination address
Traffic direction	Internal IP address destination	External IP address destination
Egress from a Google Cloud VM	Maximum possible egress is defined by the sending VM's machine type. To achieve the maximum possible egress rate, send traffic to an internal IP address associated with another Google Cloud VM in the same zone as the sending VM, in the same VPC network or a VPC network connected by VPC Network Peering.	Maximum possible egress from a single VM cannot exceed the following: 7 Gbps total for all egress flows to external IP addresses 3 Gbps per individual egress flow to an external IP address
	All egress is subject to per-project aggregate egress quotas and limits. In a situation where persistent disks compete with network egress traffic, 60% of the maximum egress bandwidth is used for persistent disks, and the remaining 40% can be used for network egress.
Ingress to a Google Cloud VM	Limited by the machine type, operating system, and network conditions. Google Cloud doesn't impose any additional limitations on ingress to an internal IP address.	Google Cloud protects each VM by limiting ingress traffic delivered to an external IP address associated with the VM. The limit is the first of the following rates encountered: 1,800,000 pps (packets per second) 20 Gbps

Egress bandwidth

Google Cloud limits outbound (egress) bandwidth on a per-VM and a per-project basis. Outbound bandwidth includes traffic sent from all of the VM's NICs and data transferred to all persistent disks connected to the VM.

The maximum possible egress bandwidth depends on the machine type of the VM. Tables for each machine type in each machine family document these numbers. For example, an n2-standard-8 VM has eight vCPUs, so its maximum possible egress bandwidth is 16 Gbps.

You can find egress bandwidth for every machine type listed on its specific machine family page:

Maximum possible egress bandwidth is not a guarantee. In addition to the machine type, egress bandwidth is impacted by factors such as the following non exhaustive list:

packet size
protocol overhead
the number of flows
Ethernet driver settings of the VM's guest OS, such as checksum offload and TCP segmentation offload (TSO)
network congestion
the destination of the packet—Google Cloud handles egress traffic from a VM differently depending on whether the outbound packet's destination address is an internal IP address or an external IP address.
In a situation where persistent disks compete with other network egress traffic, 60% of the maximum network bandwidth is given to persistent disk writes, leaving 40% for other network egress traffic. See other factors that affect performance in the Persistent Disk documentation for more details.

Egress to internal IP address destinations

From the perspective of a sending VM, Google Cloud restricts the maximum possible egress to internal IP address destinations according to the sending VM's machine type. Internal IP addresses are those in a VPC network, a different VPC network connected using VPC Network Peering, or a network connected to your VPC network using Cloud VPN or Cloud Interconnect.

The following list ranks VM-to-VM traffic, using internal IP address sources and destinations, from highest possible bandwidth to lowest:

Between VMs in the same zone
Between VMs in different zones of the same region
Between VMs in different zones in different regions

When sending traffic from a VM to an internal IP address located in a different VPC network connected using Cloud VPN tunnels, egress bandwidth is limited to the maximum data rate of a Cloud VPN tunnel.

To fully utilize the bandwidth of multiple tunnels and ECMP routing, you must use multiple TCP connections (unique 5-tuples). A five-tuple consists of a protocol, source IP address, source port, destination IP address, and destination port.

Egress to external IP address destinations

From the perspective of a sending VM, Google Cloud limits outbound traffic sent to an external IP address destination to whichever of the following rates is first reached. An external IP address is publicly routable: either an external IP address of a Google Cloud resource or an address on the internet.

7 Gbps, in total, for all packet flows and connections
3 Gbps per flow (unique five-tuple hash)

You can associate an external IP address with a Google Cloud VM in one of the following capacities:

You can assign an external IP address to the network interface of a VM.
An external forwarding rule used for protocol forwarding requires an external IP address.
The IP address of a network TCP/UDP load balancer's forwarding rule requires an external IP address.
External IP addresses are associated with a Cloud NAT gateway.

As an example, even though an n2-standard-16 instance has an egress bandwidth limit of 32 Gbps, total egress bandwidth to the internet is limited to 7 Gbps.

Per-project aggregate egress quotas and limits

Google Cloud also enforces the following for projects:

Maximum internet egress bandwidth from all VMs in each region to external IP addresses outside of Google Cloud: This maximum is defined by each region's Internet egress bandwidth quota.
Maximum egress bandwidth from all VMs in a given region to all other Google Cloud regions: Applies to traffic sent to both internal IP address destinations and external IP address destinations. This limit is calculated using internal telemetry and is unlikely to restrict inter-regional bandwidth for most projects. For questions about how to achieve your required region-to-region bandwidth, contact your sales team.

Ingress bandwidth

Google Cloud handles inbound traffic to a VM differently depending on whether the packet's destination is an internal IP address or an external IP address.

Ingress to internal IP address destinations

Google Cloud does not implement any purposeful restriction on traffic inbound to an associated internal IP address. A VM can receive as much internal traffic as its machine type, operating system, and other network conditions and resources permit. An associated internal IP address is one of the following:

The primary internal IP address of a VM's network interface
An alias IP address from an alias IP range assigned to a VM's network interface
The IP address of an internal forwarding rule used for internal protocol forwarding
The IP address of an internal TCP/UDP load balancer's forwarding rule

Ingress to external IP address destinations

Google Cloud limits inbound traffic sent to a VM's associated external IP address to whichever of the following rates that is reached first:

1,800,000 packets per second
20 Gbps

For the purposes of this limit, an associated external IP address is one of the following:

An external IP address assigned to a VM's network interface
The IP address of an external forwarding rule used for external protocol forwarding
The IP address of a network TCP/UDP load balancer's forwarding rule
Established inbound responses processed by Cloud NAT

For the last two definitions of an associated external IP address: if an external IP address is shared among multiple VMs, Google Cloud limits inbound traffic individually for each backend VM.

Receive and transmit queues

Each VM NIC is assigned a fixed number of receive and transmit queues for processing packets from the network.

Receive Queue (RX): Queue to receive packets. When the NIC receives a packet from the network, the NIC selects the descriptor for an incoming packet from the queue, processes it and hands the packet to the guest OS. If the RX queue is full and there is no buffer available to place a packet, then the packet is dropped.
Transmit Queue (TX): Queue to transmit packets. When the guest OS sends a packet, a descriptor is allocated and placed in the TX queue. The NIC then processes the descriptor and transmits the packet.

Calculating RX and TX queues

The maximum number of RX and TX queues allocated per VM is not configurable, but ethtool commands can be used to reduce the number of queues in use on Linux-based operating systems. Each VM is assigned a fixed number of RX and TX queues:

Using virtIO or a custom driver, the VM is allocated 1 queue for each vCPU with a minimum of 1 queue and maximum of 32 queues.
Using gvNIC, the VM is allocated 1 queue for every 2 vCPUs with a minimum of 1 queue and a maximum of 16 queues.

Next, each NIC is assigned a fixed number of queues calculated by dividing the number of queues assigned to the VM by the number of NICs, then rounding down to the closest whole number. For example, each NIC has five queues if a VM has 16 vCPUs and three NICs.

A larger machine size with more vCPUs has more queues allocated, meaning it can minimize the possibility of packet drops due to full buffers when processing high packet rates.