Network bandwidth

---

Google Cloud accounts for bandwidth per virtual machine (VM) instance, not per network interface (NIC) or IP address. A VM's machine type defines its maximum possible egress rate; however, you can only achieve that maximum possible egress rate in specific situations.

This page outlines expectations, which are useful when planning your deployments. It categorizes bandwidth using two dimensions:

• The traffic direction: As used on this page, traffic direction is always from the perspective of a Google Cloud VM:
  • Packets sent from a Google Cloud VM compose its egress (outbound) traffic.
  • Packets sent to a Google Cloud VM compose its ingress (inbound) traffic.

• The type of destination IP address: Google Cloud categorizes IP addresses as either internal or external:

  • An IP address within a VPC network is called an internal IP address. For example, each VM's NIC has a primary internal IP address located in a VPC network. Internal IP addresses can be any valid private or privately re-used public IP address range.
  • An IP address that's accessible from the internet is an external IP address. External IP addresses can be located in Google Cloud, such as the external IP address assigned to a VM's NIC. External IP addresses are always public IP addresses, including public IP addresses on the internet outside of Google Cloud.

  See IP addresses in the VPC documentation for precise definitions. For example, if you privately re-use a public IP address in your VPC network, that is an internal IP address, and the corresponding external IP address is no longer reachable.

Neither additional network interfaces (NICs) nor additional IP addresses per NIC increase ingress or egress bandwidth for a VM. For example, an n1-standard-8 VM with two NICs is limited to 16 Gbps total egress bandwidth, not 16 Gbps egress bandwidth per NIC.

All of the information on this page is applicable to Compute Engine VMs, as well as products that depend on Compute Engine VMs. For example, a Google Kubernetes Engine node is a Compute Engine VM.

Bandwidth summary

The following tables summarize bandwidth expectations for each type of packet destination.

Internal IP address

Traffic direction	Bandwidth expectations
Egress from a Google Cloud VM	Maximum possible egress is defined by the sending VM's machine type. N2, N2D, C2, and C2D VMs with per VM Tier_1 networking performance support egress bandwidth limits up to 100 Gbps. C3 VMs offer up to 200 Gbps egress bandwidth limits with Tier_1 networking. For details, see Configure per VM Tier_1 networking performance. To achieve the maximum possible egress rate, send traffic to an internal IP address associated with another Google Cloud VM in the same zone as the sending VM, in the same VPC network or a VPC network connected by VPC Network Peering. All egress is subject to per-project aggregate egress quotas and limits. In a situation where Persistent Disk competes with network egress traffic, 60% of the maximum egress bandwidth is used for Persistent Disk, and the remaining 40% can be used for network egress. To get the maximum egress rate, use the highest maximum transmission unit (MTU) setting. This reduces the packet-header overhead and increases payload data throughput.
Ingress to a Google Cloud VM	Limited by the machine type, operating system, and network conditions. Google Cloud doesn't impose any additional limitations on ingress to an internal IP address.

External IP address

Traffic direction	Bandwidth expectations
Egress from a Google Cloud VM	Maximum possible egress from a single VM cannot exceed the following: For all egress flows to external IP addresses, 7 Gbps total or 25 Gbps total with Tier_1 networking 3 Gbps per individual egress flow to an external IP address All egress is subject to per-project aggregate egress quotas and limits. In a situation where Persistent Disk competes with network egress traffic, 60% of the maximum egress bandwidth is used for Persistent Disk, and the remaining 40% can be used for network egress. To get the maximum egress rate, use the highest maximum transmission unit (MTU) setting. This reduces the packet-header overhead and increases payload data throughput.
Ingress to a Google Cloud VM	Google Cloud protects each VM by limiting ingress traffic delivered to an external IP address associated with the VM. The limit is the first of the following rates encountered: 1,800,000 pps (packets per second) 30 Gbps

Egress bandwidth

Google Cloud limits outbound (egress) bandwidth on a per-VM and a per-project basis. Outbound bandwidth includes traffic sent from all of the VM's NICs and data transferred to all persistent disks connected to the VM.

The standard maximum egress bandwidth depends on the machine type of the VM. The maximum egress bandwidth is generally 2 Gbps per vCPU, but there are some differences and exceptions, depending on the machine series. The following table shows the range of maximum limits for egress bandwidth for standard networking tier only, not per VM Tier_1 networking performance.

Machine series	Lowest maximum egress limit	Highest maximum egress limit
E2	1 Gbps	16 Gbps
C3	23 Gbps	100 Gbps
T2D	10 Gbps	32 Gbps
N2,C2, N2D, and C2D	10 Gbps	32 Gbps
N1 (excluding VMs with 1 vCPU)	10 Gbps	32 Gbps on Intel Skylake CPU platform 16 Gbps on CPU platforms older than Intel Skylake
N1 machine types with 1 vCPU, f1-micro, and g1-small	2 Gbps	2 Gbps
A2 and G2	Based on GPUs	Based on GPUs

You can find the maximum egress bandwidth for every machine type listed on its specific machine family page:

• General-purpose machine family
• Compute-optimized machine family
• Memory-optimized machine family
• Accelerator-optimized machine family

Maximum possible egress bandwidth is not a guarantee. In addition to the machine type, egress bandwidth is impacted by factors such as the following non-exhaustive list:

• Guest Ethernet driver—gVNIC offers better performance than the VirtIO network interface
• Packet size
• Protocol overhead
• The number of flows
• Ethernet driver settings of the VM's guest OS, such as checksum offload and TCP segmentation offload (TSO)
• Network congestion
• The destination of the packet—Google Cloud handles egress traffic from a VM differently depending on whether the outbound packet's destination address is an internal IP address or an external IP address.
• In a situation where persistent disks compete with other network egress traffic, 60% of the maximum network bandwidth is given to persistent disk writes, leaving 40% for other network egress traffic. See other factors that affect performance in the Persistent Disk documentation for more details.

You can enable per VM Tier_1 networking performance with larger general-purpose and compute-optimized machine types, which offers higher network bandwidths.

Egress to internal IP address destinations

From the perspective of a sending VM, Google Cloud restricts the maximum possible egress to internal IP address destinations according to the sending VM's machine type. Internal IP addresses are those in a VPC network, a different VPC network connected using VPC Network Peering, or a network connected to your VPC network using Cloud VPN or Cloud Interconnect.

The following list ranks VM-to-VM traffic, using internal IP address sources and destinations, from highest possible bandwidth to lowest:

• Between VMs in the same zone
• Between VMs in different zones of the same region
• Between VMs in different zones in different regions
• From a VM to Google APIs and services using Private Service Connect

When sending traffic from a VM to an internal IP address located in a different VPC network connected using Cloud VPN tunnels, egress bandwidth is limited to the maximum data rate of a Cloud VPN tunnel.

To fully utilize the bandwidth of multiple tunnels and ECMP routing, you must use multiple TCP connections (unique 5-tuples). A five-tuple consists of a protocol, source IP address, source port, destination IP address, and destination port.

Egress to external IP address destinations

From the perspective of a sending VM, Google Cloud limits outbound traffic sent to a public or external IP address destination to whichever of the following rates is first reached. An external IP address is publicly routable: either an external IP address of a Google Cloud resource or an address on the internet.

• 25 Gbps, in total, for all packet flows and connections on VMs with Tier_1 networking enabled
• 7 Gbps, in total, for all packet flows and connections on VMs without Tier_1 networking enabled
• 3 Gbps per flow (unique five-tuple hash)

You can associate an external IP address with a Google Cloud VM in one of the following capacities:

• You can assign an external IP address to the network interface of a VM.
• An external forwarding rule used for protocol forwarding requires an external IP address.
• The IP address of a network TCP/UDP load balancer's forwarding rule requires an external IP address.
• External IP addresses are associated with a Cloud NAT gateway.

As an example, even though an n2-standard-16 instance has an egress bandwidth limit of 32 Gbps, total egress bandwidth to the internet is limited to 7 Gbps.

Per-project aggregate egress quotas and limits

Google Cloud also enforces the following for projects:

• Maximum internet egress bandwidth from all VMs in each region to external IP addresses outside of Google Cloud: This maximum is defined by each region's Internet egress bandwidth quota.

• Maximum egress bandwidth from all VMs in a given region to all other Google Cloud regions: Applies to traffic sent to both internal IP address destinations and external IP address destinations. This limit is calculated using internal telemetry and is unlikely to restrict inter-regional bandwidth for most projects. For questions about how to achieve your required region-to-region bandwidth, contact your sales team.

Ingress bandwidth

Google Cloud handles inbound traffic to a VM differently depending on whether the packet's destination is an internal IP address or an external IP address.

Ingress to internal IP address destinations

Google Cloud does not implement any purposeful restriction on traffic inbound to an associated internal IP address. A VM can receive as much internal traffic as its machine type, operating system, and other network conditions and resources permit. An associated internal IP address is one of the following:

• The primary internal IP address of a VM's network interface
• An alias IP address from an alias IP range assigned to a VM's network interface
• The IP address of an internal forwarding rule used for internal protocol forwarding
• The IP address of an internal TCP/UDP load balancer's forwarding rule

Ingress to external IP address destinations

Google Cloud limits inbound traffic sent to a VM's associated external IP address to whichever of the following rates that is reached first:

• 1,800,000 packets per second
• 20 Gbps

For the purposes of this limit, an associated external IP address is one of the following:

• An external IP address assigned to a VM's network interface
• The IP address of an external forwarding rule used for external protocol forwarding
• The IP address of a network TCP/UDP load balancer's forwarding rule
• Established inbound responses processed by Cloud NAT

For the last two definitions of an associated external IP address: if an external IP address is shared among multiple VMs, Google Cloud limits inbound traffic individually for each backend VM.

Note: Depending on the machine type of your VM and other factors, actual inbound traffic might be less than 20 Gbps. Bandwidth from the internet, for example, is not covered by any SLA and is subject to network conditions.

Jumbo frames

To receive and send jumbo frames, configure the VPC network used by your VMs; set the maximum transmission unit (MTU) to a larger value, up to 8896.

Higher MTU values increase the packet size and reduce the packet-header overhead, which increases payload data throughput.

To use jumbo frames with the gVNIC driver, you must use version 1.3 or later. Not all Google Cloud public images include this driver version. For more information about operating system support for jumbo frames, see the Networking features tab on the Operating system details page.

You can download the gVNIC drivers from Github.

To manually update the gVNIC driver version in your guest OS, see Use on non-supported operating systems.

Receive and transmit queues

Each VM NIC is assigned a number of receive and transmit queues for processing packets from the network.

• Receive Queue (RX): Queue to receive packets. When the NIC receives a packet from the network, the NIC selects the descriptor for an incoming packet from the queue, processes it and hands the packet to the guest OS over a packet queue attached to a vCPU core via interrupt. If the RX queue is full and there is no buffer available to place a packet, then the packet is dropped. This can typically happen if an application is over-utilizing a vCPU core that is also attached to the selected packet queue.
• Transmit Queue (TX): Queue to transmit packets. When the guest OS sends a packet, a descriptor is allocated and placed in the TX queue. The NIC then processes the descriptor and transmits the packet.

Default queue allocation

Unless you explicitly assign queue counts for NICs, you can model the algorithm Google Cloud uses to assign a fixed number of RX and TX queues per NIC in this way:

1. Use one of the following methods, depending on your network interface type:

   • VirtIO: Divide the number of vCPUs by the number of NICs, and discard any remainder —[number of vCPUs/number of NICs].
   • gVNIC: Divide the number of vCPUs by the number of NICs, and then divide the result by 2 and discard any remainder — [number of vCPUs/number of NICs/2].

   This calculation always results in a whole number (not a fraction).

2. If the calculated number is less than 1, assign each NIC one queue instead.

3. Determine if the calculated number is greater than the maximum number of queues per NIC. The maximum number of queues per NIC depends on the driver type:

   • Using virtIO or a custom driver, the maximum number of queues per NIC is 32. If the calculated number is greater than 32, ignore the calculated number, and assign each NIC 32 queues instead.
   • Using gVNIC, the maximum number of queues per NIC is 16. If the calculated number is greater than 16, ignore the calculated number, and assign each NIC 16 queues instead.

The following examples show how to calculate the default number of queues:

• If a VM uses VirtIO and has 16 vCPUs and 4 NICs, the calculated number is [16/4] = 4. Google Cloud assigns each NIC four queues.

• If a VM using gVNIC and has 128 vCPUs and two NICs, the calculated number is [128/2/2] = 32. Google Cloud assigns each NIC the maximum number of queues per NIC possible. Google Cloud assigns 16 queues per NIC.

On Linux systems, you can use ethtool to configure a NIC with fewer queues than the number of queues Google Cloud assigns per NIC.

Custom queue allocation

Instead of the Default queue allocation, you can assign a custom queue count (total of both RX and TX) to each NIC when you create a new VM by using by using the Compute Engine API.

The number of custom queues you specify must adhere to the following rules:

• The minimum queue count you can assign per NIC is one.

• The maximum queue count you can assign is the lower of the vCPU count or the per NIC maximum queue count, based on the driver type:

  • Using virtIO or a custom driver, the maximum queue count is 32.
  • Using gVNIC, the maximum queue count is 16.

• If you assign custom queue counts to all NICs of the VM, the sum of your queue count assignments must be less than or equal to the number of vCPUs assigned to the VM instance.

Examples

• If a VM has 8 vCPUs and 3 NICs, the maximum queue count for the VM is the number of vCPUS, or 8. You can assign 1 queue to nic0, 4 queues to nic1, and 3 queues to nic2. In this example, you cannot subsequently assign 4 queues to nic2 while keeping the other two NIC queue assignments because the sum of assigned queues cannot exceed the number of vCPUs (8).

• If a VM has 96 vCPUs and 2 NICs, you can assign both NICs up to 32 queues each when using the virtIO driver, or up to 16 queues each when using the gVNIC driver. In this example, the sum of assigned queues is always less than the number of vCPUs.

It's also possible to assign a custom queue count for only some NICs, letting Google Cloud assign queues to the remaining NICs. The number of queues you can assign per NIC is still subject to rules above. You can model the feasibility of your configuration, and, if your configuration is possible, the number of queues Google Cloud assigns to the remaining NICs with this process:

1. Calculate the sum of queues for the NICs using custom queue assignment. For an example VM with 20 vCPUs and 6 NICs, suppose you assign nic0 5 queues, nic1 6 queues, nic2 4 queues, and let Google Cloud assign queues for nic3, nic4, and nic5. In this example, the sum of custom-assigned queues is 5+6+4 = 15.

2. Subtract the sum of custom-assigned queues from the number of vCPUs. If the difference is not at least equal to the number of remaining NICs for which Google Cloud must assign queues, Google Cloud returns an error. Continuing with the example VM of 20 vCPUs and a sum of 15 custom-assigned queues, Google Cloud has 20-15 = 5 queues left to assign to the remaining NICs (nic3, nic4, nic5).

3. Divide the difference from the previous step by the number of remaining NICs and discard any remainder —⌊(number of vCPUs - sum of assigned queues)/(number of remaining NICs)⌋. This calculation always results in a whole number (not a fraction) that is at least equal to one because of the constraint explained in the previous step. Google Cloud assigns each remaining NICs a queue count matching the calculated number as long as the calculated number is not greater than the maximum number of queues per NIC. The maximum number of queues per NIC depends on the driver type:

   • Using virtIO or a custom driver, if the calculated number of queues for each remaining NIC is greater than 32, Google Cloud assigns each remaining NIC 32 queues.
   • Using gVNIC, if the calculated number of queues for each remaining NIC is greater than 16, Google Cloud assigns each remaining NIC 16 queues.

API

Create a VM with specific queue counts for NICs using the instances.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances
{
  "name": "VM_NAME",
  "machineType": "machineTypes/MACHINE_TYPE"
  "networkInterfaces": [
      {
        "network": string,
        "subnetwork": string,
        "networkIP": string,
        "name": string,
        "queueCount": "QUEUE_SIZE",
        ....
      ],
      } ],
 }
 

Replace the following:

• PROJECT_ID: ID of the project to create the VM in
• ZONE: zone to create the VM in
• MACHINE_TYPE: machine type, predefined or custom, for the new VM
• VM_NAME: name of the new VM
• QUEUE_SIZE: Number of queues for the NIC, subject to the rules discussed in this section.

Queue allocations and changing the machine type

VMs are created with a default queue allocation, or you can assign a custom queue count to each virtual network interface card (vNIC) when you create a new VM by using the Compute Engine API. The default or custom vNIC queue assignments are only set when creating a VM. If your VM hs vNICs which use default queue counts, you can change its machine type. If the machine type that you are changing to has a different number of vCPUs, the default queue counts for your VM are recalculated based on the new machine type.

If your VM has vNICs which use custom, non-default queue counts, you can change the machine type by using the Google Cloud CLI or Compute Engine API to update the instance properties. The conversion succeeds if the resulting VM supports the same queue count per vNIC as the original VM. For VMs that use the VirtIO-Net interface and have a custom queue count that is higher than 16 per vNIC, you can't change the machine type to a third generation machine type, which uses only gVNIC. Instead, you can migrate your VM to a third generation machine type by following the instructions in Migrate your workload from an existing VM to a new VM.

What's next

• Machine types
• Virtual machine instances
• Creating and starting a VM instance
• Configuring per VM Tier_1 networking performance
• Quickstart using a Linux VM
• Quickstart using a Windows VM