Configure per VM Tier_1 networking performance

Compute Engine lets you select a high-bandwidth per VM Tier_1 networking performance configuration for certain general-purpose and compute-optimized virtual machine (VM) instances. VMs with Tier_1 networking configurations are especially useful for large, distributed compute workloads with lots of heavy internode communications, such as high performance computing (HPC), machine learning (ML), and deep learning (DL).

Combining these high throughput VMs with high-performance local SSD storage is beneficial for I/O-intensive, flash-optimized databases.

Before you begin

Review the pricing for per VM Tier_1 networking performance at Tier_1 higher bandwidth network pricing.
If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

Select the tab for how you plan to use the samples on this page:
Console

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
1. Install the Google Cloud CLI, then initialize it by running the following command:
```
gcloud init
```
  Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
2. Set a default region and zone.
REST

To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

Requirements and limitations

Compute Engine is the only product area supporting Tier_1 networking.
Tier_1 networking is supported on N2, N2D, C2, C2D, C3, C3D, and M3 VMs that have at least 30 vCPUs.
Tier_1 networking requires VMs that use the gVNIC virtual network driver and a gVNIC-compatible OS or custom image.
C3 and C3D VMs require gVNIC driver version 1.3 or later to deliver the best Tier_1 networking performance. Make sure the operating system image that you use fully supports Tier_1 networking. Fully supported OS images include the updated gVNIC driver. You can update the gVNIC driver on images that don't have the latest version.
Purchasable stock keeping units (SKUs) for Tier_1 networking are excluded from committed use discounts.
Large C3 and C3D VMs may encounter NUMA-related bottlenecks when bandwidth is pushed beyond 100 Gbps. Depending on your application architecture, you might need to control for thread and interrupt placement. On Linux, guest OS features such as Receive Flow Steering (RFS) can help address this issue. Ensure that your applications are NUMA-tuned to maximize your performance.
On C3, C3D, and H3 VMs that use Microsoft Windows, the gVNIC driver can achieve up to 85 Gbps of network bandwidth, for both the default network and Tier_1 networking.

Bandwidth tiers

The egress bandwidth limit represents the maximum possible amount of data per unit of time (for example, gigabits per second, or Gbps) that Google Cloud allows a VM to emit from its network interfaces (NICs). The egress bandwidth includes data transferred to all Persistent Disk and Hyperdisk volumes attached to the VM.

Note the following about bandwidth limits:

The default bandwidth limit ranges from 10 Gbps to 200 Gbps, depending on the machine type and VM size.
Tier_1 networking increases the maximum egress bandwidth limit for VMs. The maximum egress bandwidth limit ranges from 50 Gbps to 200 Gbps, depending on the size and machine type of your VM.
The actual egress bandwidth is always less than or equal to the egress bandwidth limit.

To achieve the highest possible egress bandwidth, all of the following must be true:

The sending and receiving VMs must be in the same zone.
The VMs must have NICs in the same VPC network or in VPC networks connected by VPC Network Peering.
Packets sent between the VMs must use internal IP address destinations.
The VPC network used by the VMs uses the highest maximum transmission unit (MTU) setting. A higher MTU reduces the packet-header overhead and thus increases payload data throughput.

For a complete discussion about egress and ingress bandwidth limits, see Network bandwidth.

General-purpose N2 VMs

The following table describes the egress bandwidth limits for N2 VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
2	10 Gbps	Not applicable (N/A)	7 Gbps	N/A
4	10 Gbps	N/A	7 Gbps	N/A
8	16 Gbps	N/A	7 Gbps	N/A
16	32 Gbps	N/A	7 Gbps	N/A
32	32 Gbps	50 Gbps	7 Gbps	25 Gbps
48	32 Gbps	50 Gbps	7 Gbps	25 Gbps
64	32 Gbps	75 Gbps	7 Gbps	25 Gbps
80	32 Gbps	100 Gbps	7 Gbps	25 Gbps
96	32 Gbps	100 Gbps	7 Gbps	25 Gbps
128	32 Gbps	100 Gbps	7 Gbps	25 Gbps

General-purpose N2 (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2 VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
32-62	32 Gbps	50 Gbps	7 Gbps	25 Gbps
64-78	32 Gbps	75 Gbps	7 Gbps	25 Gbps
80-128	32 Gbps	100 Gbps	7 Gbps	25 Gbps

General-purpose N2D VMs

The following table describes the egress bandwidth limits for N2D VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
2	10 Gbps	Not applicable (N/A)	7 Gbps	N/A
4	10 Gbps	N/A	7 Gbps	N/A
8	16 Gbps	N/A	7 Gbps	N/A
16	32 Gbps	N/A	7 Gbps	N/A
32	32 Gbps	N/A	7 Gbps	N/A
48	32 Gbps	50 Gbps	7 Gbps	25 Gbps
64	32 Gbps	50 Gbps	7 Gbps	25 Gbps
80	32 Gbps	50 Gbps	7 Gbps	25 Gbps
96	32 Gbps	100 Gbps	7 Gbps	25 Gbps
128	32 Gbps	100 Gbps	7 Gbps	25 Gbps
224	32 Gbps	100 Gbps	7 Gbps	25 Gbps

General-purpose N2D (custom size shapes) VMs

The following table describes the egress bandwidth limits for custom-sized N2D VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
48-94	32 Gbps	50 Gbps	7 Gbps	25 Gbps
96	32 Gbps	100 Gbps	7 Gbps	25 Gbps

General-purpose C3 VMs

The following table describes the egress bandwidth limits for C3 VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
4	23 Gbps	N/A	7 Gbps	N/A
8	23 Gbps	N/A	7 Gbps	N/A
22	23 Gbps	N/A	7 Gbps	N/A
44	32 Gbps	50 Gbps	7 Gbps	25 Gbps
88	62 Gbps	100 Gbps	7 Gbps	25 Gbps
176	100 Gbps	200 Gbps	7 Gbps	25 Gbps

General-purpose C3D VMs

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
4	20 Gbps	N/A	7 Gbps	N/A
8	20 Gbps	N/A	7 Gbps	N/A
16	20 Gbps	N/A	7 Gbps	N/A
30	20 Gbps	50 Gbps	7 Gbps	25 Gbps
60	40 Gbps	75 Gbps	7 Gbps	25 Gbps
90	60 Gbps	100 Gbps	7 Gbps	25 Gbps
180	100 Gbps	150 Gbps	7 Gbps	25 Gbps
360	100 Gbps	200 Gbps	7 Gbps	25 Gbps

Compute-optimized C2 VMs

The following table describes the egress bandwidth limits for C2 VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
4	10 Gbps	Not applicable (N/A)	7 Gbps	N/A
8	16 Gbps	N/A	7 Gbps	N/A
16	32 Gbps	N/A	7 Gbps	N/A
30	32 Gbps	50 Gbps	7 Gbps	7 Gbps
60	32 Gbps	100 Gbps	7 Gbps	7 Gbps

Compute-optimized C2D VMs

The following table describes the egress bandwidth limits for C2D VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
2	10 Gbps	Not applicable (N/A)	7 Gbps	N/A
4	10 Gbps	N/A	7 Gbps	N/A
8	16 Gbps	N/A	7 Gbps	N/A
16	32 Gbps	N/A	7 Gbps	N/A
32	32 Gbps	50 Gbps	7 Gbps	7 Gbps
56	32 Gbps	50 Gbps	7 Gbps	7 Gbps
112	32 Gbps	100 Gbps	7 Gbps	7 Gbps

Compute-optimized H3 VMs

The following table describes the egress bandwidth limits for H3 VMs.

vCPUs	Internal IP	External IP
88	200 Gbps	1 Gbps

Memory-optimized M3 VMs

The following table describes the egress bandwidth limits for M3 VMs.

vCPUs	Internal IP	Tier_1 Internal IP	External IP	Tier_1 External IP
32	32 Gbps	N/A	7 Gbps	N/A
64	32 Gbps	50 Gbps	7 Gbps	25 Gbps
128	32 Gbps	100 Gbps	7 Gbps	25 Gbps

Configure a VM with Tier_1 networking

You can enable Tier_1 networking during VM creation, if the VM uses the gVNIC network interface. You can also edit a VM to add or remove Tier_1 networking, provided the VM was created with the gVNIC interface.

Optionally, you can also enable faster network packet processing with DPDK to run performance-intensive applications on a VM that uses Tier_1 networking.

Create VMs and containers that use Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or REST to add Tier_1 networking to a new VM or container.

Permissions required for this task

To perform this task, you must have the following permissions:

compute.instances.create on the project
compute.instances.updateShieldedVmConfig if you plan to create a Shielded VM instance and you want to be able to change any of the Shielded VM settings
compute.networks.use on the project if using a legacy network
compute.subnetworks.use either on the whole project or on the chosen subnet (VPC networks)
compute.networks.useExternalIp on the project if you need to assign an external IP address (either ephemeral or static) to the instance using a legacy network
compute.subnetworks.useExternalIp either on the whole project or on the chosen subnet if you need to assign an external IP address (either ephemeral or static) to the instance using a VPC network
compute.addresses.use on the project if specifying a static address in the project
compute.instances.setMetadata if setting metadata
compute.instances.setTags on the instance if setting tags
compute.instances.setLabels on the instance if setting labels
compute.instances.setServiceAccount on the instance if setting service account
compute.images.useReadOnly on the image if creating a new root persistent disk
compute.disks.create on the project if creating a new root persistent disk with this instance
compute.disks.useReadOnly on the disk if attaching an existing persistent disk in read-only mode
compute.disks.use on the disk if attaching an existing disk in read/write mode
compute.disks.setLabels on the disk if setting labels
compute.snapshots.create on the project to create a new snapshot if creating an instance from a snapshot
compute.snapshots.useReadOnly on the snapshot if creating an instance from a snapshot
compute.instanceTemplates.useReadOnly on the instance template if creating instance from instance template

Console

In the Google Cloud console, go to the VM instances page.

Go to VM instances
Select your project.
Click Create instance.
Specify a Name for your VM. For more information, see Resource naming convention.
Select a region and zone that supports Tier_1 networking.
Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, C3, or C3D VM.
- Click the General purpose tab before selecting N2, N2D, C3, or C3D from the Series drop-down menu.
- Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.
To select a gVNIC compatible operating system, in the Boot disk section, click Change, and then select a supported operating system or use the Custom Images tab to select a custom image.
Optional. In the Firewall section, choose your firewall rules.
To change your networking properties, click the heading Advanced options to expand the section.
Click Networking and then do the following:
1. In the Network interface card drop-down menu, select gVNIC.
2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.
3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.
Click Create.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Use the gcloud compute instances create command to create a VM with a gVNIC virtual network driver. Use the --network-performance-configs flag and the --network-interface flag to configure a network performance setting for a VM. If you don't specify these flags the VM is created with the default network performance configuration.

To create a VM running container images, use the gcloud compute instances create-with-container command.
```
gcloud compute instances create VM_NAME  \
    --image=OS_IMAGE  \
    --machine-type=MACHINE_TYPE  \
    --network-performance-configs=total-egress-bandwidth-tier=TIER_1  \
    --network-interface=nic-type=GVNIC
```
Replace the following:
- VM_NAME: the name of the VM
- OS_IMAGE: an image that supports gVNIC image
- MACHINE_TYPE: a machine type that supports a high-bandwidth configuration
For example:
```
   gcloud compute instances create instance-1 \
       --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
       --network-interface=nic-type=GVNIC \
       --image-family=rocky-linux-8-optimized-gcp \
       --image-project=rocky-linux-cloud \
       --machine-type=n2-standard-32
   
```

REST

Call the Compute Engine API instances.insert method to create a VM with a high-bandwidth network configuration. Within the request body:

Set the networkPerformanceConfig parameters to totalEgressBandwidthTier and TIER_1.
Set the networkInterface parameters to nicType and GVNIC.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances

{
  "name": VM_NAME,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": TIER_1
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
    ]
  }

Replace the following:

PROJECT_ID: your project ID
ZONE: the zone where you want to create the VM
VM_NAME: the name of the VM

Update a VM to include Tier_1 networking

Refer to the Updating instance properties documentation to ensure you are meeting all the requirements to successfully update your VM. Use the Google Cloud console, the Google Cloud CLI or REST to update a VM.

You can modify an existing VM to change the network configuration to include or exclude per VM Tier_1 networking performance. Your VM must already have a gVNIC interface associated with it; you can't edit your VM to add a network interface. To update the network configuration, you must stop and restart the VM.

Permissions required for this task

To perform this task, you must have the following permissions:

compute.instances.update on the VM instance.
Permission to use the resources that you want to modify on the instance, for example compute.instances.updateNetworkInterface.

These permissions are included in several existing Identity and Access Management (IAM) roles.

Console

In the Google Cloud console, go to the VM instances page.

Go to VM instances
Select your project.
Click the name of the VM that you wish to modify.
Stop the VM.
Select Edit.
If your VM was originally configured with a gVNIC card, select the Enable per VM Tier_1 networking performance checkbox to add per VM Tier_1 networking performance, or deselect the checkbox to remove this feature from your VM.
Save your changes.
Restart your VM.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

Export your VM's information to a .yaml file using the gcloud compute instances export command.
```
gcloud compute instances export VM_NAME \
    --zone=ZONE --destination=PATH_TO_FILE
```
Replace the following:
- VM_NAME: the name of the VM
- ZONE: the name of the zone where the VM is located
- PATH_TO_FILE: a file name with a .yaml extension.
  
  For example:
```
gcloud compute instances export instance-1 \
    --zone=europe-west1-c --destination=test-file.yaml
```
Open the Cloud Shell Editor, or the editor of your choice.
Open the .yaml file you created.
Scroll to the end of the file and add the following information:
```
networkPerformanceConfig:
  totalEgressBandwidthTier: TIER_1
```
Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.
Use the gcloud compute instance update-from-file command to update the VM with the changes in the file.
```
gcloud compute instances update-from-file VM_NAME \
    --source=PATH_TO_FILE \
    --most-disruptive-allowed-action=RESTART
```
Replace the following:
- VM_NAME: the name of the VM
- PATH_TO_FILE: your .yaml file name
The --most-disruptive-allowed-action=RESTART flag setting automatically restarts your VM with the updated configuration.

REST

Call the instances.update method to modify the network configuration.

PUT https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID?most_disruptive_allowed_action=RESTART

{
  "networkPerformanceConfig":{
      "totalEgressBandwidthTier": "TIER_1"
  },
...
}

Setting totalEgressBandwidthTier to TIER_1 adds Tier_1 networking. Setting it to DEFAULT removes the configuration.

Replace the following:

PROJECT_ID: your project ID
ZONE: the zone where your VM resides
RESOURCE_ID: the name of your VM

The most_disruptive_allowed_action=RESTART query parameter automatically restarts your VM with the updated configuration.

Verify high-bandwidth configuration in a VM

Use the Google Cloud console, the Google Cloud CLI or REST to generate a description of an existing VM or an existing VM running container images to verify the VM's bandwidth tier.

Console

In the Google Cloud console, go to the VM instances page.

Go to VM instances
Select your project and click Continue.
Click on the instance name to see its configuration details and see if the VM uses per VM Tier_1 networking performance.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

Use the gcloud compute instances describe command to check if your VM uses per VM Tier_1 networking performance.

For example:
```
  gcloud compute instances describe VM_NAME  \
      --format="text(name, networkPerformanceConfig)"
  
```
The output is similar to the following:
```
   name: instance-1
   networkPerformanceConfig.totalEgressBandwidthTier:TIER_1
  
```
If the output shows the value DEFAULT, then Tier_1 networking isn't enabled.

REST

Call the Compute Engine API instances.get method to view the network configuration.

GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/RESOURCE_ID/

Replace the following:

PROJECT_ID: your project name
ZONE: the zone where your VM resides
RESOURCE_ID: the name of your VM

The output should contain the following lines:

{
  "name": RESOURCE_ID,
    "description": string,
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
    },
    ...
  }

If the output shows the value DEFAULT, then Tier_1 networking is not enabled.

Create an instance template with Tier_1 networking

Use the Google Cloud console, the Google Cloud CLI or REST to create an instance template with per VM Tier_1 networking performance. Refer to the Creating an instance template documentation to ensure you are meeting all the requirements to create your VM instance template.

Permissions required for this task

To perform this task, you must have the following permissions:

All the permissions required to call the instanceTemplates.insert method.

These permissions are included in several existing Identity and Access Management (IAM) roles.

Console

In the Google Cloud console, go to the Instance templates page.

Go to Instance templates
Click Create instance template.
Enter values for the following fields, or accept the default values.
Specify a Name for your instance template. For more information, see Resource naming convention.
Select a region and zone that supports Tier_1 networking.
Select a Machine configuration for your VM. To create a VM with Tier_1 networking, you must select an N2, N2D, C2, C2D, C3, or C3D VM.
- Click the General purpose tab before selecting N2, N2D, C3, or C3D from the Series drop-down menu.
- Click the Compute optimized tab before selecting C2 or C2D from the Series drop-down menu.
In the Machine type drop-down menu, choose a machine type that aligns with the bandwidth tier size requirements.
In the Boot disk section, click Change, and then select a gVNIC-compatible or custom image.
Optional. In the Firewall section, choose your firewall rules.
Expand the Advanced options section to change your networking properties.
Click Networking and then do the following:
1. In the Network interface card drop-down menu, select gVNIC.
  
  Note: The value "-" in the Network interface card drop down menu indicates that the NIC type can be either gVNIC or VirtioNet (VirtIO) depending on the machine family type. For some machine family types, only gVNIC is available for the network interface card. If both gVNIC and VirtIO are available for a VM, choose gVNIC instead of using the default, which is VirtIO.
2. Under Network bandwidth, select the Enable per VM Tier_1 networking performance checkbox.
3. If your VM has multiple NICs or you use IPv6 addresses, configure your Network interfaces.
Click Create.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

Use the gcloud compute instance-templates create command with the --network-performance-configs, and the --network-interface flags.

  gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \
      --image=OS_IMAGE \
      --machine-type=MACHINE_TYPE \
      --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
      --network-interface=nic-type=GVNIC

Replace the following:

INSTANCE_TEMPLATE_NAME: the name of your instance template
OS_IMAGE: an image that supports gVNIC image
MACHINE_TYPE: a machine type that supports Tier_1 networking, as described in Bandwidth tiers.

For example:

  gcloud compute instance-templates create instance-template-1 \
      --image-family=rocky-linux-8-optimized-gcp \
      --image-project=rocky-linux-cloud \
      --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
      --machine-type=n2-standard-32 \
      --network-interface=nic-type=GVNIC

REST

Call the Compute Engine API instanceTemplates.insert method. Within the request body, set the networkPerformanceConfig parameter to totalEgressBandwidthTier and TIER_1. Set the networkInterfaces parameter to nicType and GVNIC.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instancesTemplates

{
  "name": "INSTANCE_TEMPLATE_NAME",
  "properties": {
    "machineType": "zones/ZONE/machineTypes/MACHINE_TYPE",
    ...
    "networkPerformanceConfig": {
        "totalEgressBandwidthTier": "TIER_1"
  },
    "networkInterfaces": [
      {
        "nicType": "GVNIC"
    },
    ...
  }

Replace the following:

PROJECT_ID: your project name
INSTANCE_TEMPLATE_NAME: your instance template name
ZONE: the zone where your VM is located
MACHINE_TYPE: the machine type of the VM
RESOURCE_ID: the name of your VM

Benchmark a higher bandwidth configuration

You can run a benchmark test to check your VM's performance with per VM Tier_1 networking performance. Be sure to remove the benchmarking resources you created during testing to avoid unexpected resource charges.

What's next

High-bandwidth pricing