Configure per VM Tier_1 networking performance


Compute Engine lets you select a high-bandwidth per VM Tier_1 networking performance configuration for certain general-purpose and compute-optimized virtual machine (VM) instances. VMs with Tier_1 networking configurations are especially useful for large, distributed compute workloads with lots of heavy internode communications, such as high performance computing (HPC), machine learning (ML), and deep learning (DL).

Combining these high throughput VMs with high-performance local SSD storage is beneficial for I/O-intensive, flash-optimized databases.

Before you begin

  • Review the pricing for per VM Tier_1 networking performance at Tier_1 higher bandwidth network pricing.
  • If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. Install the Google Cloud CLI, then initialize it by running the following command:

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command:

      gcloud init

Required roles

To get the permissions that you need to configure a VM to use per VM Tier_1 networking performance, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access.

These predefined roles contain the permissions required to configure a VM to use per VM Tier_1 networking performance. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to configure a VM to use per VM Tier_1 networking performance:

  • To create VMs:
    • compute.instances.create on the project
    • To use a custom image to create the VM: compute.images.useReadOnly on the image
    • To use a snapshot to create the VM: compute.snapshots.useReadOnly on the snapshot
    • To use an instance template to create the VM: compute.instanceTemplates.useReadOnly on the instance template
    • To assign a legacy network to the VM: compute.networks.use on the project
    • To specify a static IP address for the VM: compute.addresses.use on the project
    • To assign an external IP address to the VM when using a legacy network: compute.networks.useExternalIp on the project
    • To specify a subnet for the VM: compute.subnetworks.use on the project or on the chosen subnet
    • To assign an external IP address to the VM when using a VPC network: compute.subnetworks.useExternalIp on the project or on the chosen subnet
    • To set VM instance metadata for the VM: compute.instances.setMetadata on the project
    • To set tags for the VM: compute.instances.setTags on the VM
    • To set labels for the VM: compute.instances.setLabels on the VM
    • To set a service account for the VM to use: compute.instances.setServiceAccount on the VM
    • To create a new disk for the VM: compute.disks.create on the project
    • To attach an existing disk in read-only or read-write mode: compute.disks.use on the disk
    • To attach an existing disk in read-only mode: compute.disks.useReadOnly on the disk
  • To update a VM to include Tier_1 networking:
    • compute.instances.update
    • Permission to use the resources that you want to modify on the instance, for example compute.instances.updateNetworkInterface
  • To create an instance template with Tier_1 networking: All the permissions required to call the instanceTemplates.insert method

You might also be able to get these permissions with custom roles or other predefined roles.

Limitations

  • Compute Engine is the only product area supporting Tier_1 networking.
  • Tier_1 networking is supported on N2, N2D, C2, C2D, C3, C3D, M3, and Z3 (Preview) VMs that have at least 30 vCPUs.
  • Tier_1 networking requires VMs that use the gVNIC virtual network driver and a gVNIC-compatible OS or custom image.
  • C3, C3D, Z3 (Preview), and H3 VMs require gVNIC driver version 1.3 or later to deliver the best Tier_1 networking performance. Make sure the operating system image that you use fully supports Tier_1 networking. Fully supported OS images include the updated gVNIC driver. You can update the gVNIC driver on images that don't have the latest version.
  • Purchasable stock keeping units (SKUs) for Tier_1 networking are excluded from committed use discounts.
  • Large C3, C3D, and Z3 (Preview) VMs may encounter NUMA-related bottlenecks when bandwidth is pushed beyond 100 Gbps. Depending on your application architecture, you might need to control for thread and interrupt placement. On Linux, guest OS features such as Receive Flow Steering (RFS) can help address this issue. Ensure that your applications are NUMA-tuned to maximize your performance.
  • On C3, C3D, and H3 VMs that use Microsoft Windows, the gVNIC driver can achieve up to 85 Gbps of network bandwidth, for both the default network and Tier_1 networking.

Bandwidth tiers

The egress bandwidth limit represents the maximum possible amount of data per unit of time (for example, gigabits per second, or Gbps) that Google Cloud allows a VM to emit from its network interfaces (NICs). The egress bandwidth includes data transferred to all Persistent Disk and Google Cloud Hyperdisk volumes attached to the VM.

Note the following about bandwidth limits:

  • The default bandwidth limit ranges from 10 Gbps to 200 Gbps, depending on the machine type and VM size.
  • Tier_1 networking increases the maximum egress bandwidth limit for VMs. The maximum egress bandwidth limit ranges from 50 Gbps to 200 Gbps, depending on the size and machine type of your VM.
  • The actual egress bandwidth is always less than or equal to the egress bandwidth limit.

To achieve the highest possible egress bandwidth, all of the following must be true:

  • The sending and receiving VMs must be in the same zone.
  • The VMs must have NICs in the same VPC network or in VPC networks connected by VPC Network Peering.
  • Packets sent between the VMs must use internal IP address destinations.
  • The VPC network used by the VMs uses the highest maximum transmission unit (MTU) setting. A higher MTU reduces the packet-header overhead and thus increases payload data throughput.

For a complete discussion about egress and ingress bandwidth limits, see Network bandwidth.

General-purpose N2 VMs

The following table describes the egress bandwidth limits for N2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps 50 Gbps 7 Gbps 25 Gbps
48 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64 32 Gbps 75 Gbps 7 Gbps 25 Gbps
80 32 Gbps 100 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps
128 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2 (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
32-62 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64-78 32 Gbps 75 Gbps 7 Gbps 25 Gbps
80-128 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2D VMs

The following table describes the egress bandwidth limits for N2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps N/A 7 Gbps N/A
48 32 Gbps 50 Gbps 7 Gbps 25 Gbps
64 32 Gbps 50 Gbps 7 Gbps 25 Gbps
80 32 Gbps 50 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps
128 32 Gbps 100 Gbps 7 Gbps 25 Gbps
224 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose N2D (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
48-94 32 Gbps 50 Gbps 7 Gbps 25 Gbps
96 32 Gbps 100 Gbps 7 Gbps 25 Gbps

General-purpose C3 VMs

The following table describes the egress bandwidth limits for C3 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
4 23 Gbps N/A 7 Gbps N/A
8 23 Gbps N/A 7 Gbps N/A
22 23 Gbps N/A 7 Gbps N/A
44 32 Gbps 50 Gbps 7 Gbps 25 Gbps
88 62 Gbps 100 Gbps 7 Gbps 25 Gbps
176 100 Gbps 200 Gbps 7 Gbps 25 Gbps

General-purpose C3D VMs

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
4 20 Gbps N/A 7 Gbps N/A
8 20 Gbps N/A 7 Gbps N/A
16 20 Gbps N/A 7 Gbps N/A
30 20 Gbps 50 Gbps 7 Gbps 25 Gbps
60 40 Gbps 75 Gbps 7 Gbps 25 Gbps
90 60 Gbps 100 Gbps 7 Gbps 25 Gbps
180 100 Gbps 150 Gbps 7 Gbps 25 Gbps
360 100 Gbps 200 Gbps 7 Gbps 25 Gbps

Compute-optimized C2 VMs

The following table describes the egress bandwidth limits for C2 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
4 10 Gbps Not applicable (N/A) 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
30 32 Gbps 50 Gbps 7 Gbps 7 Gbps
60 32 Gbps 100 Gbps 7 Gbps 7 Gbps

Compute-optimized C2D VMs

The following table describes the egress bandwidth limits for C2D VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
2 10 Gbps Not applicable (N/A) 7 Gbps N/A
4 10 Gbps N/A 7 Gbps N/A
8 16 Gbps N/A 7 Gbps N/A
16 32 Gbps N/A 7 Gbps N/A
32 32 Gbps 50 Gbps 7 Gbps 7 Gbps
56 32 Gbps 50 Gbps 7 Gbps 7 Gbps
112 32 Gbps 100 Gbps 7 Gbps 7 Gbps

Compute-optimized H3 VMs

The following table describes the egress bandwidth limits for H3 VMs.

vCPUs Internal IP External IP
88 200 Gbps 1 Gbps

Memory-optimized M3 VMs

The following table describes the egress bandwidth limits for M3 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
32 32 Gbps N/A 7 Gbps N/A
64 32 Gbps 50 Gbps 7 Gbps 25 Gbps
128 32 Gbps 100 Gbps 7 Gbps 25 Gbps

Storage-optimized Z3 VMs

The following table describes the egress bandwidth limits for Z3 VMs.

vCPUs Internal IP Tier_1 Internal IP External IP Tier_1 External IP
88 62 Gbps 100 Gbps 7 Gbps 25 Gbps
176 100 Gbps 200 Gbps 7 Gbps 25 Gbps

Configure a VM with Tier_1 networking

You can enable Tier_1 networking during VM creation, if the VM uses the gVNIC network interface. You can also edit a VM to add or remove Tier_1 networking, provided the VM was created with the gVNIC interface.

Optionally, you can also enable faster network packet processing with DPDK to run performance-intensive applications on a VM that uses Tier_1 networking.

Create VMs and containers that use Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or REST to add Tier_1 networking to a new VM or container.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project.

  3. Click Create instance.

  4. Specify a Name for your VM. For more information, see Resource naming convention.

  5. Select a region and zone that supports Tier_1 networking.

  6. Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, C3, C3D, or Z3 (Preview) VM.

    • Click the General purpose tab before selecting N2, N2D, C3, or C3D from the Series drop-down menu.
    • Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
    • Click the Storage optimized tab before selecting Z3 from the Series drop-down menu.
  7. In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.

  8. To select a gVNIC compatible operating system, in the Boot disk section, click Change, and then select a supported operating system or use the Custom Images tab to select a custom image.

  9. Optional. In the Firewall section, choose your firewall rules.

  10. To change your networking properties, click the heading Advanced options to expand the section.

  11. Click Networking and then do the following:

    1. In the Network interface card drop-down menu, select gVNIC.
    2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.
    3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.
  12. Click Create.

gcloud

    In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  1. Use the gcloud compute instances create command to create a VM with a gVNIC virtual network driver. Use the --network-performance-configs flag and the --network-interface flag to configure a network performance setting for a VM. If you don't specify these flags the VM is created with the default network performance configuration.

    To create a VM running container images, use the gcloud compute instances create-with-container command.

    gcloud compute instances create VM_NAME  \
        --image=OS_IMAGE  \
        --machine-type=MACHINE_TYPE  \
        --network-performance-configs=total-egress-bandwidth-tier=TIER_1  \
        --network-interface=nic-type=GVNIC
    

    Replace the following:

    For example:

       gcloud compute instances create instance-1 \
           --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
           --network-interface=nic-type=GVNIC \
           --image-family=rocky-linux-8-optimized-gcp \
           --image-project=rocky-linux-cloud \
           --machine-type=n2-standard-32
       

REST

Call the Compute Engine API instances.insert method to create a VM with a high-bandwidth network configuration. Within the request body:

  • Set the networkPerformanceConfig parameters to totalEgressBandwidthTier and TIER_1.
  • Set the networkInterface parameters to nicType and GVNIC.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

{
  "name": VM_NAME,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": TIER_1
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
    ]
  }

Replace the following:

  • PROJECT_ID: your project ID
  • ZONE: the zone where you want to create the VM
  • VM_NAME: the name of the VM

Update a VM to include Tier_1 networking

Refer to the Updating instance properties documentation to ensure you are meeting all the requirements to successfully update your VM. Use the Google Cloud console, the Google Cloud CLI or REST to update a VM.

You can modify an existing VM to change the network configuration to include or exclude per VM Tier_1 networking performance. Your VM must already have a gVNIC interface associated with it; you can't edit your VM to add a network interface. To update the network configuration, you must stop and restart the VM.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project.

  3. Click the name of the VM that you want to modify.

  4. Stop the VM.

  5. Select Edit.

  6. If your VM was originally configured with a gVNIC card, select the Enable per VM Tier_1 networking performance checkbox to add per VM Tier_1 networking performance, or deselect the checkbox to remove this feature from your VM.

  7. Save your changes.

  8. Restart your VM.

gcloud

    In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  1. Export your VM's information to a .yaml file using the gcloud compute instances export command.

    gcloud compute instances export VM_NAME \
        --zone=ZONE --destination=PATH_TO_FILE
    

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the name of the zone where the VM is located
    • PATH_TO_FILE: a filename with a .yaml extension.

      For example:

      gcloud compute instances export instance-1 \
          --zone=europe-west1-c --destination=test-file.yaml
      
  2. Open the Cloud Shell Editor, or the editor of your choice.
  3. Open the .yaml file you created.
  4. Go to the end of the file and add the following information:

    networkPerformanceConfig:
      totalEgressBandwidthTier: TIER_1
    

    Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.

  5. Use the gcloud compute instance update-from-file command to update the VM with the changes in the file.

    gcloud compute instances update-from-file VM_NAME \
        --source=PATH_TO_FILE \
        --most-disruptive-allowed-action=RESTART
    

    Replace the following:

    • VM_NAME: the name of the VM
    • PATH_TO_FILE: your .yaml filename

    The --most-disruptive-allowed-action=RESTART flag setting automatically restarts your VM with the updated configuration.

REST

Call the instances.update method to modify the network configuration.

PUT https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID?most_disruptive_allowed_action=RESTART

{
  "networkPerformanceConfig":{
      "totalEgressBandwidthTier": "TIER_1"
  },
...
}

Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.

Replace the following:

  • PROJECT_ID: your project ID
  • ZONE: the zone where your VM resides
  • RESOURCE_ID: the name of your VM

The most_disruptive_allowed_action=RESTART query parameter automatically restarts your VM with the updated configuration.

Verify high-bandwidth configuration in a VM

Use the Google Cloud console, the Google Cloud CLI or REST to generate a description of an existing VM or an existing VM running container images to verify the VM's bandwidth tier.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Select your project and click Continue.

  3. Click the VM name to see its configuration details and see if the VM uses per VM Tier_1 networking performance.

gcloud

    In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  1. Use the gcloud compute instances describe command to check if your VM uses per VM Tier_1 networking performance.

    For example:

      gcloud compute instances describe VM_NAME  \
          --format="text(name, networkPerformanceConfig)"
      

    The output is similar to the following:

       name: instance-1
       networkPerformanceConfig.totalEgressBandwidthTier:TIER_1
      

    If the output shows the value DEFAULT, then Tier_1 networking isn't enabled.

REST

Call the Compute Engine API instances.get method to view the network configuration.

GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID/

Replace the following:

  • PROJECT_ID: your project name
  • ZONE: the zone where your VM resides
  • RESOURCE_ID: the name of your VM

The output should contain the following lines:

{
  "name": RESOURCE_ID,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
    },
    ...
  }

If the output shows the value DEFAULT, then Tier_1 networking is not enabled.

Create an instance template with Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or REST to create an instance template with per VM Tier_1 networking performance. Refer to the Creating an instance template documentation to ensure you are meeting all the requirements to create your VM instance template.

Console

  1. In the Google Cloud console, go to the Instance templates page.

    Go to Instance templates

  2. Click Create instance template.

  3. Enter values for the following fields, or accept the default values.

  4. Specify a Name for your instance template. For more information, see Resource naming convention.

  5. Select a region and zone that supports Tier_1 networking.

  6. Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, C3, C3D, or Z3 (Preview) VM.

    • Click the General purpose tab before selecting N2, N2D, C3, or C3D from the Series drop-down menu.
    • Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
    • Click the Storage optimized tab before selecting Z3 from the Series drop-down menu.
  7. In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.

  8. In the Boot disk section, click Change, and then select a gVNIC-compatible or custom image.

  9. Optional. In the Firewall section, choose your firewall rules.

  10. Expand the Advanced options section to change your networking properties.

  11. Click Networking and then do the following:

    1. In the Network interface card drop-down menu, select gVNIC.

    2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.

    3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.

  12. Click Create.

gcloud

    In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  1. Use the gcloud compute instance-templates create command with the --network-performance-configs, and the --network-interface flags.

      gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \
          --image=OS_IMAGE \
          --machine-type=MACHINE_TYPE \
          --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
          --network-interface=nic-type=GVNIC
      

    Replace the following:

    • INSTANCE_TEMPLATE_NAME: the name of your instance template
    • OS_IMAGE: an image that supports gVNIC image
    • MACHINE_TYPE: a machine type that supports Tier_1 networking, as described in Bandwidth tiers.

    For example:

      gcloud compute instance-templates create instance-template-1 \
          --image-family=rocky-linux-8-optimized-gcp \
          --image-project=rocky-linux-cloud \
          --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
          --machine-type=n2-standard-32 \
          --network-interface=nic-type=GVNIC
      

REST

Call the Compute Engine API instanceTemplates.insert method. Within the request body, set the networkPerformanceConfig parameter to totalEgressBandwidthTier and TIER_1. Set the networkInterfaces parameter to nicType and GVNIC.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instancesTemplates

{
  "name": "INSTANCE_TEMPLATE_NAME",
  "properties": {
    "machineType": "zones/ZONE/machineTypes/MACHINE_TYPE",
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
  }

Replace the following:

  • PROJECT_ID: your project name
  • INSTANCE_TEMPLATE_NAME: your instance template name
  • ZONE: the zone where your VM is located
  • MACHINE_TYPE: the machine type of the VM
  • RESOURCE_ID: the name of your VM

Benchmark a higher bandwidth configuration

You can run a benchmark test to check your VM's performance with per VM Tier_1 networking performance. Be sure to remove the benchmarking resources you created during testing to avoid unexpected resource charges.

What's next