Configure per VM Tier_1 networking performance


Compute Engine lets you select a high-bandwidth per VM Tier_1 networking performance configuration for certain general-purpose and compute-optimized virtual machine (VM) instances. VMs with Tier_1 networking configurations are especially useful for large, distributed compute workloads with lots of heavy internode communications, such as high performance computing (HPC), machine learning (ML), and deep learning (DL).

Combining these high throughput VMs with high-performance local SSD storage is beneficial for I/O-intensive, flash-optimized databases.

Before you begin

Requirements and limitations

  • Compute Engine is the only product area supporting Tier_1 networking.
  • Tier_1 networking is only supported on N2, N2D, C2, C2D, or C3 VMs that have at least 30 vCPUs.
  • Tier_1 networking requires VMs that use the gVNIC virtual network driver and a gVNIC-compatible OS or custom image.
  • C3 VMs require gVNIC driver version 1.3 or later to deliver the best Tier_1 networking performance. Make sure the operating system image that you use fully supports Tier_1 networking. Fully supported OS images include the updated gVNIC driver.
  • Purchasable stock keeping units (SKUs) for Tier_1 networking are excluded from committed use discounts.
  • Large C3 VMs may encounter NUMA-related bottlenecks when bandwidth is pushed beyond 100 Gbps. Depending on application architecture, you may need to control for thread and interrupt placement. On Linux, guest OS features such as Receive Flow Steering (RFS) can help address this issue. Ensure that your applications are NUMA-tuned to maximize your performance.
  • On C3 VMs that use Microsoft Windows, the gVNIC driver can achieve up to 85 Gbps of network bandwidth, for both the default network and Tier_1 networking.

Bandwidth tiers

The egress bandwidth limit represents the maximum possible amount of data per unit of time (for example, gigabits per second, or Gbps) that Google Cloud allows a VM to emit from its network interfaces (NICs). The egress bandwidth includes data transferred to all Persistent Disk and Hyperdisk volumes attached to the VM.

Note the following about bandwidth limits:

  • The default bandwidth limit ranges from 10 Gbps to 100 Gbps, depending on the machine type and VM size.
  • Tier_1 networking increases the maximum egress bandwidth limit for VMs. The maximum egress bandwidth limit ranges from 50 Gbps to 200 Gbps, depending on the size and machine type of your VM.
  • The actual egress bandwidth is always less than or equal to the egress bandwidth limit.

To achieve the highest possible egress bandwidth, all of the following must be true:

  • The sending and receiving VMs must be in the same zone.
  • The VMs must have NICs in the same VPC network or in VPC networks connected by VPC Network Peering.
  • Packets sent between the VMs must use internal IP address destinations.
  • The VPC network used by the VMs uses the highest maximum transmission unit (MTU) setting. A higher MTU reduces the packet-header overhead and thus increases payload data throughput.

For a complete discussion about egress and ingress bandwidth limits, see Network bandwidth.

General-purpose N2 VMs

The following table describes the egress bandwidth limits for N2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps 50 Gbps 7 Gbps 25 Gbps
48 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64 32 Gbps 75 Gbps 7 Gbps 25 Gbps
80 32 Gbps 100 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps
128 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2 (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
32-62 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64-78 32 Gbps 75 Gbps 7 Gbps 25 Gbps
80-128 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2D VMs

The following table describes the egress bandwidth limits for N2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps N/A 7 Gbps N/A
48 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64 32 Gbps 50 Gbps 7 Gbps 25 Gbps
80 32 Gbps 50 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps
128 32 Gbps 100 Gbps 7 Gbps 25 Gbps
224 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2D (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
48-94 32 Gbps 50 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose C3 VMs

The following table describes the egress bandwidth limits for C3 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
4 23 Gbps N/A 7 Gbps N/A
8 23 Gbps N/A 7 Gbps N/A
22 23 Gbps N/A 7 Gbps N/A
44 32 Gbps 50 Gbps 7 Gbps 25 Gbps
88 62 Gbps 100 Gbps 7 Gbps 25 Gbps
176 100 Gbps 200 Gbps 7 Gbps 25 Gbps

Compute-optimized C2 VMs

The following table describes the egress bandwidth limits for C2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
4 10 Gbps Not applicable (N/A) 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
30 32 Gbps 50 Gbps 7 Gbps 7 Gbps
60 32 Gbps 100 Gbps 7 Gbps 7 Gbps

Compute-optimized C2D VMs

The following table describes the egress bandwidth limits for C2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps 50 Gbps 7 Gbps 7 Gbps
56 32 Gbps 50 Gbps 7 Gbps 7 Gbps
112 32 Gbps 100 Gbps 7 Gbps 7 Gbps

Configure a VM with Tier_1 networking

You can enable Tier_1 networking during VM creation, if the VM uses the gVNIC network interface. You can also edit a VM to add or remove Tier_1 networking, provided the VM was created with the gVNIC interface.

Create VMs and containers that use Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or the Compute Engine API to add Tier_1 networking to a new VM or container.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project.

  3. Click Create instance.

  4. Specify a Name for your VM. For more information, see Resource naming convention.

  5. Select a region and zone that supports Tier_1 networking.

  6. Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, or C3 VM.

    • Click the General purpose tab before selecting N2, N2D, or C3 from the Series drop-down menu.
    • Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
  7. In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.

  8. To select a gVNIC compatible operating system, in the Boot disk section, click Change, and then select a supported operating system or use the Custom Images tab to select a custom image.

  9. Optional. In the Firewall section, choose your firewall rules.

  10. To change your networking properties, click the heading Advanced options to expand the section.

  11. Click Networking and then do the following:

    1. In the Network interface card drop-down menu, select gVNIC.
    2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.
    3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.
  12. Click Create.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Use the gcloud compute instances create command to create a VM with a gVNIC virtual network driver. Use the --network-performance-configs flag and the --network-interface flag to configure a network performance setting for a VM. If you don't specify these flags the VM is created with the default network performance configuration.

    To create a VM running container images, use the gcloud compute instances create-with-container command.

    gcloud compute instances create VM_NAME  \
        --image=OS_IMAGE  \
        --machine-type=MACHINE_TYPE  \
        --network-performance-configs=total-egress-bandwidth-tier=TIER_1  \
        --network-interface=nic-type=GVNIC
    

    Replace the following:

    For example:

       gcloud compute instances create instance-1 \
           --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
           --network-interface=nic-type=GVNIC \
           --image-family=rocky-linux-8-optimized-gcp \
           --image-project=rocky-linux-cloud \
           --machine-type=n2-standard-32
       

API

Call the Compute Engine API instances.insert method to create a VM with a high-bandwidth network configuration. Within the request body:

  • Set the networkPerformanceConfig parameters to totalEgressBandwidthTier and TIER_1.
  • Set the networkInterface parameters to nicType and GVNIC.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

{
  "name": VM_NAME,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": TIER_1
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
  }

Replace the following:

  • PROJECT_ID: your project ID
  • ZONE: the zone where you want to create the VM
  • VM_NAME: the name of the VM

Update a VM to include Tier_1 networking

Refer to the Updating instance properties documentation to ensure you are meeting all the requirements to successfully update your VM. Use the Google Cloud console, the Google Cloud CLI or the Compute Engine API to update a VM.

You can modify an existing VM to change the network configuration to include or exclude per VM Tier_1 networking performance. Your VM must already have a gVNIC interface associated with it; you can't edit your VM to add a network interface. To update the network configuration, you must stop and restart the VM.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project.

  3. Click the name of the VM that you wish to modify.

  4. Stop the VM.

  5. Select Edit.

  6. If your VM was originally configured with a gVNIC card, select the Enable per VM Tier_1 networking performance checkbox to add per VM Tier_1 networking performance, or deselect the checkbox to remove this feature from your VM.

  7. Save your changes.

  8. Restart your VM.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Export your VM's information to a .yaml file using the gcloud compute instances export command.

    gcloud compute instances export VM_NAME \
        --zone=ZONE --destination=PATH_TO_FILE
    

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the name of the zone where the VM is located
    • PATH_TO_FILE: a file name with a .yaml extension.

      For example:

      gcloud compute instances export instance-1 \
          --zone=europe-west1-c --destination=test-file.yaml
      
  3. Open the Cloud Shell Editor, or the editor of your choice.
  4. Open the .yaml file you created.
  5. Scroll to the end of the file and add the following information:

    networkPerformanceConfig:
    totalEgressBandwidthTier: TIER_1
    

    Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.

  6. Use the gcloud compute instance update-from-file command to update the VM with the changes in the file.

    gcloud compute instances update-from-file VM_NAME \
        --source=PATH_TO_FILE \
        --most-disruptive-allowed-action=RESTART
    

    Replace the following:

    • VM_NAME: the name of the VM
    • PATH_TO_FILE: your .yaml file name

    The --most-disruptive-allowed-action=RESTART flag setting automatically restarts your VM with the updated configuration.

API

Call the instances.update method to modify the network configuration.

PUT https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID?most_disruptive_allowed_action=RESTART

{
  "networkPerformanceConfig":{
  "totalEgressBandwidthTier": "TIER_1"
  },
...
}

Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.

Replace the following:

  • PROJECT_ID: your project ID
  • ZONE: the zone where your VM resides
  • RESOURCE_ID: the name of your VM

The most_disruptive_allowed_action=RESTART query parameter automatically restarts your VM with the updated configuration.

Verify high-bandwidth configuration in a VM

Use the Google Cloud console, the Google Cloud CLI or the Compute Engine API to generate a description of an existing VM or an existing VM running container images to verify the VM's bandwidth tier.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project and click Continue.

  3. Click on the instance name to see its configuration details and see if the VM uses per VM Tier_1 networking performance.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Use the gcloud compute instances describe command to check if your VM uses per VM Tier_1 networking performance.

    For example:

      gcloud compute instances describe VM_NAME  \
          --format="text(name, networkPerformanceConfig)"
      

    The output is similar to the following:

       name: instance-1
       networkPerformanceConfig.totalEgressBandwidthTier:TIER_1
      

    If the output shows the value DEFAULT, then Tier_1 networking isn't enabled.

API

Call the Compute Engine API instances.get method to view the network configuration.

GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID/

Replace the following:

  • PROJECT_ID: your project name
  • ZONE: the zone where your VM resides
  • RESOURCE_ID: the name of your VM

The output should contain the following lines:

{
  "name": RESOURCE_ID,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
    },
    ...
  }

If the output shows the value DEFAULT, then Tier_1 networking is not enabled.

Create an instance template with Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or the Compute Engine API to create an instance template with per VM Tier_1 networking performance. Refer to the Creating an instance template documentation to ensure you are meeting all the requirements to create your VM instance template.

Console

  1. In the Google Cloud console, go to the Instance templates page.

    Go to Instance templates

  2. Click Create instance template.

  3. Enter values for the following fields, or accept the default values.

  4. Specify a Name for your instance template. For more information, see Resource naming convention.

  5. Select a region and zone that supports Tier_1 networking.

  6. Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, or C3 VM.

    • Click the General purpose tab before selecting N2, N2D, or C3 from the Series drop-down menu.
    • Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
  7. In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.

  8. In the Boot disk section, click Change, and then select a gVNIC-compatible or custom image.

  9. Optional. In the Firewall section, choose your firewall rules.

  10. Expand the Advanced options section to change your networking properties.

  11. Click Networking and then do the following:

    1. In the Network interface card drop-down menu, select gVNIC.

    2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.

    3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.

  12. Click Create.

gcloud

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. Use the gcloud compute instance-templates create command with the --network-performance-configs, and the --network-interface flags.

      gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \
          --image=OS_IMAGE \
          --machine-type=MACHINE_TYPE \
          --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
          --network-interface=nic-type=GVNIC
      

    Replace the following:

    • INSTANCE_TEMPLATE_NAME: the name of your instance template
    • OS_IMAGE: an image that supports gVNIC image
    • MACHINE_TYPE: a machine type that supports Tier_1 networking, as described in Bandwidth tiers.

    For example:

      gcloud compute instance-templates create instance-template-1 \
          --image-family=rocky-linux-8-optimized-gcp \
          --image-project=rocky-linux-cloud \
          --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
          --machine-type=n2-standard-32 \
          --network-interface=nic-type=GVNIC
      

API

Call the Compute Engine API instanceTemplates.insert method. Within the request body, set the networkPerformanceConfig parameter to totalEgressBandwidthTier and TIER_1. Set the networkInterfaces parameter to nicType and GVNIC.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instancesTemplates

{
  "name": "INSTANCE_TEMPLATE_NAME",
  "properties": {
    "machineType": "zones/ZONE/machineTypes/MACHINE_TYPE",
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
  }

Replace the following:

  • PROJECT_ID: your project name
  • INSTANCE_TEMPLATE_NAME: your instance template name
  • ZONE: the zone where your VM is located
  • MACHINE_TYPE: the machine type of the VM
  • RESOURCE_ID: the name of your VM

Benchmark a higher bandwidth configuration

You can run a benchmark test to check your VM's performance with per VM Tier_1 networking performance. Be sure to remove the benchmarking resources you created during testing to avoid unexpected resource charges.

What's next